HPC-driven computational reproducibility in numerical relativity codes: A use case study with IllinoisGRMHD (2307.01343v2)
Abstract: Reproducibility of results is a cornerstone of the scientific method. Scientific computing encounters two challenges when aiming for this goal. Firstly, reproducibility should not depend on details of the runtime environment, such as the compiler version or computing environment, so results are verifiable by third-parties. Secondly, different versions of software code executed in the same runtime environment should produce consistent numerical results for physical quantities. In this manuscript, we test the feasibility of reproducing scientific results obtained using the IllinoisGRMHD code that is part of an open-source community software for simulation in relativistic astrophysics, the Einstein Toolkit. We verify that numerical results of simulating a single isolated neutron star with IllinoisGRMHD can be reproduced, and compare them to results reported by the code authors in 2015. We use two different supercomputers: Expanse at SDSC, and Stampede2 at TACC. By compiling the source code archived along with the paper on both Expanse and Stampede2, we find that IllinoisGRMHD reproduces results published in its announcement paper up to errors comparable to round-off level changes in initial data parameters. We also verify that a current version of IlliinoisGRMHD reproduces these results once we account for bug fixes which has occurred since the original publication
- 2022 FAIR for research software (FAIR4RS) WG URL https://www.rd-alliance.org/groups/fair-4-research-software-fair4rs-wg
- Martinez P A 2020 Towards FAIR principles for research software URL https://eresearchnz.figshare.com/articles/presentation/Towards_FAIR_principles_for_research_software/11929617/1
- RDA | research data sharing without barriers https://www.rd-alliance.org/ URL https://www.rd-alliance.org/
- FORCE11 the future of research communications and e-scholarship https://force11.org/ URL https://force11.org/
- 2022 <ReSA> research software alliance URL https://www.researchsoft.org/
- 2022 The software sustainability institute URL https://software.ac.uk/
- UK reproducibility network https://www.ukrn.org/ URL https://www.ukrn.org/
- Abbott B P e a (LIGO Scientific Collaboration and Virgo Collaboration) 2016 Phys. Rev. Lett. 116(6) 061102 URL https://link.aps.org/doi/10.1103/PhysRevLett.116.061102
- Spec: Spectral einstein code URL https://www.black-holes.org/code/SpEC.html
- Bruegmann B, Tichy W and Jansen N 2004 Phys. Rev. Lett. 92 211101 (Preprint gr-qc/0312112)
- Cactus Computational Toolkit URL http://www.cactuscode.org/
- SDSC https://www.sdsc.edu/support/user_guides/expanse.html URL https://www.sdsc.edu/support/user_guides/expanse.html
- TACC https://portal.tacc.utexas.edu/user-guides/stampede2#table1 URL https://portal.tacc.utexas.edu/user-guides/stampede2#table1
- Etienne Z 2022 IllinoisGRMHD2015 simulation software package and raw dataset URL https://doi.org/10.5281/zenodo.7545717
- Etienne Z https://bitbucket.org/zach_etienne/wvuthorns.git URL https://bitbucket.org/zach_etienne/wvuthorns.git
- Etienne Z http://astro.phys.wvu.edu/zetienne/ILGRMHD/ URL http://astro.phys.wvu.edu/zetienne/ILGRMHD/
- SimFactory: Herding numerical simulations URL http://simfactory.org/
- About Snaps https://snapcraft.io/about URL https://snapcraft.io/about
- Jacobsen D M and Canon R S 2015
- Kurtzer G M, Sochat V and Bauer M W 2017 PLOS ONE 12 1–20 URL https://doi.org/10.1371/journal.pone.0177459
- Canon R S and Younge A 2019 A case for portability and reproducibility of HPC containers 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) pp 49–54
- 2021 SpEC: Spectral Einstein Code URL https://www.black-holes.org/code/SpEC.html
- Einstein Toolkit: Open software for relativistic astrophysics URL http://einsteintoolkit.org/