Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts (2310.08677v2)

Published 12 Oct 2023 in cs.LG and cs.AI

Abstract: Geometric deep learning (GDL) has gained significant attention in scientific fields, for its proficiency in modeling data with intricate geometric structures. However, very few works have delved into its capability of tackling the distribution shift problem, a prevalent challenge in many applications. To bridge this gap, we propose GeSS, a comprehensive benchmark designed for evaluating the performance of GDL models in scientific scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics, materials science to biochemistry, and encapsulate a broad spectrum of distribution shifts including conditional, covariate, and concept shifts. Furthermore, we study three levels of information access from the out-of-distribution (OOD) test data, including no OOD information, only unlabeled OOD data, and OOD data with a few labels. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting. A thorough analysis of the evaluation results is provided, poised to illuminate insights for GDL researchers and domain practitioners who are to use GDL in their applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (85)
  1. A common tracking software project, 2021.
  2. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019, 2022.
  3. A comprehensive guide to the physics and usage of pythia 8.3. SciPost Physics Codebases, pp.  008, 2022.
  4. Large-scale benchmark of exchange–correlation functionals for the determination of electronic band gaps of solids. Journal of chemical theory and computation, 15(9):5069–5079, 2019.
  5. Iglue: A benchmark for transfer learning across modalities, tasks, and languages. In International Conference on Machine Learning, pp.  2370–2392. PMLR, 2022.
  6. Invariant rationalization. In International Conference on Machine Learning, pp.  1448–1458. PMLR, 2020.
  7. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, 11(10):6059–6072, 2021.
  8. Learning causally invariant representations for out-of-distribution generalization on graphs. Advances in Neural Information Processing Systems, 35:22131–22148, 2022.
  9. Diffdock: Diffusion steps, twists, and turns for molecular docking. In International Conference on Learning Representations (ICLR 2023), 2023.
  10. Environment inference for invariant learning. In International Conference on Machine Learning, pp.  2189–2200. PMLR, 2021.
  11. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  12. Graph neural networks for particle tracking and reconstruction. In Artificial intelligence for high energy physics, pp.  387–436. World Scientific, 2022.
  13. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428, 2019.
  14. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970–1981, 2020.
  15. Benchmarking graph neural networks for materials chemistry. npj Computational Materials, 7(1):84, 2021.
  16. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1–37, 2014.
  17. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096–2030, 2016.
  18. Rlsbench: Domain adaptation under relaxed label shift. In International Conference on Machine Learning, pp.  10879–10928. PMLR, 2023.
  19. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2019.
  20. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
  21. Good: A graph out-of-distribution benchmark. Advances in Neural Information Processing Systems, 35:2059–2073, 2022.
  22. Thomas A Halgren. Mmff vi. mmff94s option for energy minimization studies. Journal of computational chemistry, 20(7):720–729, 1999.
  23. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
  24. Roger Highfield. Large hadron collider: Thirteen ways to change the world. The Daily Telegraph. London. Retrieved, pp.  10–10, 2008.
  25. Transfer learning on large datasets for the accurate prediction of material properties. arXiv preprint arXiv:2303.03000, 2023.
  26. The robustness limits of sota vision models to natural variation. Transactions on Machine Learning Research, 2023.
  27. Drugood: Out-of-distribution (ood) dataset curator and benchmark for ai-aided drug discovery–a focus on affinity prediction problems with noise annotations. arXiv preprint arXiv:2201.09637, 2022.
  28. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
  29. Modeling the data-generating process is necessary for out-of-distribution generalization. In The Eleventh International Conference on Learning Representations, 2022.
  30. Jan Kieseler. Object condensation: one-stage grid-free multi-object reconstruction in physics detectors, graph, and image data. The European Physical Journal C, 80:1–12, 2020.
  31. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  32. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp.  5637–5664. PMLR, 2021.
  33. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp.  5815–5826. PMLR, 2021.
  34. Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8:31, 2013.
  35. Evaluating out-of-distribution performance on document image classifiers. Advances in Neural Information Processing Systems, 35:11673–11685, 2022.
  36. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations, 2022.
  37. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In The Eleventh International Conference on Learning Representations, 2022.
  38. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35:24529–24542, 2022.
  39. Structural re-weighting improves graph domain adaptation. In International Conference on Machine Learning, pp.  21778–21793. PMLR, 2023.
  40. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations, 2021.
  41. Pileup mitigation at the large hadron collider with graph neural networks. The European Physical Journal Plus, 134(7):333, 2019.
  42. Interpretable geometric deep learning via learnable randomness injection. In The Eleventh International Conference on Learning Representations, 2022.
  43. Density functional theory. Photosynthesis research, 102:443–453, 2009.
  44. Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 33(3):1065–1076, 1962.
  45. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
  46. Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19(2):3, 2000.
  47. Jet tagging via particle clouds. Physical Review D, 101(5):056019, 2020.
  48. Machine-learning-assisted materials discovery using failed experiments. Nature, 533(7601):73–76, 2016.
  49. Machine learning at the energy and intensity frontiers of particle physics. Nature, 560(7716):41–48, 2018.
  50. Better informed distance geometry: using what we know to improve conformation generation. Journal of chemical information and modeling, 55(12):2562–2574, 2015.
  51. Machine learning the quantum-chemical properties of metal–organic frameworks for accelerated materials discovery. Matter, 4(5):1578–1597, 2021.
  52. High-throughput predictions of metal–organic framework electronic properties: theoretical challenges, graph neural networks, and data exploration. npj Computational Materials, 8(1):112, 2022.
  53. Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pp.  832–837, 1956.
  54. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
  55. Extending the wilds benchmark for unsupervised adaptation. In International Conference on Learning Representations (ICLR), 2022.
  56. Breeds: Benchmarks for subpopulation shift. In International Conference on Learning Representations, 2021.
  57. E (n) equivariant graph neural networks. In International conference on machine learning, pp.  9323–9332. PMLR, 2021.
  58. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
  59. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pp.  9377–9388. PMLR, 2021.
  60. Graph neural networks in particle physics. Machine Learning: Science and Technology, 2(2):021001, 2020.
  61. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp.  443–450. Springer, 2016.
  62. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations, 2021.
  63. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.
  64. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts. ACS Catalysis, 13(5):3066–3084, 2023.
  65. Applications of machine learning in drug discovery and development. Nature reviews Drug discovery, 18(6):463–477, 2019.
  66. Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988–999, 1999.
  67. Comenet: Towards complete and efficient message passing for 3d molecular graphs. Advances in Neural Information Processing Systems, 35:650–664, 2022.
  68. Mixup for node and graph classification. In Proceedings of the Web Conference 2021, pp.  3663–3674, 2021.
  69. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1–12, 2019.
  70. David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988.
  71. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2021.
  72. Discovering invariant rationales for graph neural networks. In International Conference on Learning Representations, 2021.
  73. In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. In International Conference on Learning Representations, 2020.
  74. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018.
  75. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073, 2022a.
  76. Learning substructure invariance for out-of-distribution molecular representations. Advances in Neural Information Processing Systems, 35:12964–12978, 2022b.
  77. Wild-time: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 35:10309–10324, 2022a.
  78. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp.  25407–25437. PMLR, 2022b.
  79. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7947–7958, 2022.
  80. Molecular scaffold analysis of natural products databases in the public domain. Chemical biology & drug design, 80(5):717–724, 2012.
  81. Benchmarking test-time adaptation against distribution shifts in image classification. arXiv preprint arXiv:2307.03133, 2023.
  82. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
  83. Domain adaptation under target and conditional shift. In International conference on machine learning, pp.  819–827. PMLR, 2013.
  84. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  16259–16268, 2021.
  85. Calculation of semiconductor band gaps with the m06-l density functional. The Journal of chemical physics, 130(7), 2009.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now