Reproducing GW150914: the first observation of gravitational waves from a binary black hole merger (2010.07244v2)

Abstract: In 2016, LIGO and Virgo announced the first observation of gravitational waves from a binary black hole merger, known as GW150914. To establish the confidence of this detection, large-scale scientific workflows were used to measure the event's statistical significance. They used code written by the LIGO/Virgo and were executed on the LIGO Data Grid. The codes are publicly available, but there has not yet been an attempt to directly reproduce the results, although several analyses have replicated the analysis, confirming the detection. We attempt to reproduce the result presented in the GW150914 discovery paper using publicly available code on the Open Science Grid. We show that we can reproduce the main result but we cannot exactly reproduce the LIGO analysis as the original data set used is not public. We discuss the challenges we encountered and make recommendations for scientists who wish to make their work reproducible.

• The paper reproduces the original >5.1σ statistical significance finding of GW150914 using open science data and PyCBC.
• It adapts LIGO workflows to the Open Science Grid, addressing challenges in software versioning and data calibration.
• The study highlights the importance of comprehensive documentation and containerization to enhance reproducibility in future research.

Reproducing GW150914: The First Observation of Gravitational Waves from a Binary Black Hole Merger

The paper "Reproducing GW150914: The first observation of gravitational waves from a binary black hole merger" documents an effort to reproduce the landmark detection of gravitational waves (GWs), GW150914, observed by the Advanced Laser Interferometer Gravitational-Wave Observatory (LIGO) and Virgo. Authored by Duncan A. Brown, Karan Vahi, Michela Taufer, Von Welch, and Ewa Deelman, the paper focuses on the challenges and methodologies involved in achieving reproducibility using publicly available data and codes.

Key Findings and Methodologies

The primary objective of this paper is to reproduce the statistical significance analysis of the GW150914 event presented in the original LIGO paper using the PyCBC search for gravitational waves. This analysis was critical in establishing that the observed signal was detected with a significance greater than $5.1\sigma$. The authors successfully reproduced the main result, albeit with some small, but noticeable, differences in the search background due to the unavailability of the original dataset, which was proprietary to LIGO.

Computational and Data Challenges

The paper outlines various challenges encountered during the reproduction effort:

1. Software Versioning: The original analysis used specific versions of the PyCBC code and associated libraries. The authors noted that the versions were neither documented in the discovery paper nor the technical paper, requiring them to refer to the notes from the original analysis team.
2. Data Availability: The proprietary nature of the LIGO Data Grid and the calibration differences between the original data (C01 calibration) and the publicly released data (C02 calibration) were significant obstacles. The analysis leveraged the Gravitational Wave Open Science Center (GWOSC) for public data access.
3. Workflow Execution: The original workflows were designed to run on the LIGO Data Grid, a closed computational environment. The reproduction effort adapted these workflows to run on the Open Science Grid (OSG). This adaptation involved modifying the workflow to work with the newer versions of the Pegasus workflow management system and other software components.
4. Resource Management: The computational requirements were substantial, with the workflow comprising nearly 42,000 tasks that required approximately 22 years of cumulative job wall time. The execution also demanded significant memory resources, and handling job failures and memory rescaling was crucial for successful completions.

Implications and Recommendations

The authors provide several recommendations for ensuring reproducibility in future scientific endeavors:

• Comprehensive Documentation: Ensure that version-specific details of codes and exact computations are thoroughly documented.
• Data and Metadata Accessibility: Provide access to the original data sets and any relevant metadata used in the analyses.
• Containerization: Use containerized environments, such as Singularity or Docker, to capture the complete software stack and dependencies.
• Long-term Archival: Maintain repositories for all relevant codes, data sets, and workflows to preserve the integrity of scientific findings over time.

Conclusion

The paper underscores the importance of reproducibility in computational science, particularly in high-impact fields such as gravitational-wave astronomy. The successful reproduction of the GW150914 analysis, despite the hurdles, serves as a testament to the robustness of the scientific workflow management systems and the potential of open data initiatives. Looking ahead, this work emphasizes the need for continued efforts to make scientific results reproducible and accessible, thereby strengthening the foundation of empirical research and collaborative advancements in AI and other computationally intensive fields.