Employing Continuous Integration inspired workflows for benchmarking of scientific software -- a use case on numerical cut cell quadrature

Abstract: In the field of scientific computing, one often finds several alternative software packages (with open or closed source code) for solving a specific problem. These packages sometimes even use alternative methodological approaches, e.g., different numerical discretizations. If one decides to use one of these packages, it is often not clear which one is the best choice. To make an informed decision, it is necessary to measure the performance of the alternative software packages for a suitable set of test problems, i.e. to set up a benchmark. However, setting up benchmarks ad-hoc can become overwhelming as the parameter space expands rapidly. Very often, the design of the benchmark is also not fully set at the start of some project. For instance, adding new libraries, adapting metrics, or introducing new benchmark cases during the project can significantly increase complexity and necessitate laborious re-evaluation of previous results. This paper presents a proven approach that utilizes established Continuous Integration tools and practices to achieve high automation of benchmark execution and reporting. Our use case is the numerical integration (quadrature) on arbitrary domains, which are bounded by implicitly or parametrically defined curves or surfaces in 2D or 3D.