Comparing transit spectroscopy pipelines at the catalogue level: evidence for systematic differences

Abstract: The challenge of inconsistent results from different data pipelines, even when starting from identical data, is a recognized concern in exoplanetary science. As we transition into the James Webb Space Telescope (JWST) era and prepare for the ARIEL space mission, addressing this issue becomes paramount because of its implications on our understanding of exoplanets. Although comparing pipeline results for individual exoplanets has become more common, this study is the first to compare pipeline results at the catalogue level. We present a comprehensive framework to statistically compare the outcomes of data analysis reduction on a population of exoplanets and we leverage the large number of observations conducted using the same instrument configured with HST-WFC3. We employ three independent pipelines: Iraclis, EXCALIBUR, and CASCADe. Our combined findings reveal that these pipelines, despite starting from the same data and planet system parameters, yield substantially different spectra in some cases. However, the most significant manifestations of pipeline differences are observed in the compositional trends of the resulting exoplanet catalogues. We conclude that pipeline-induced differences lead to biases in the retrieved information, which are not reflected in the retrieved uncertainties. Our findings underscore the critical need to confront these pipeline differences to ensure the reproducibility, accuracy, and reliability of results in exoplanetary research. Our results demonstrate the need to understand the potential for population-level bias that pipelines may inject, which could compromise our understanding of exoplanets as a class of objects.