Using script generators for pipeline prototyping (2112.10050v1)

Abstract: Fully automated astronomical data calibration and imaging pipelines are difficult to develop without a good prototyping method which permits to bridge the time between observatory commissioning and the moment when the special features and possible problems of the data and their processing are fully understood. In this paper I present a method which has worked well for the ALMA observatory and which is sufficiently general to be transferable to most other projects. In short, the idea is to use a three-level data analysis software design (scriptable toolkit, script generator, automated pipeline) and a corresponding timing of the software development which is ramping up the effort in three stages starting at the beginning of construction, at the beginning of commissioning, and at the end of commissioning respectively. The important design pattern which I would like to underline here is the use of script generators as prototypes for the automated pipeline.