A scalable transient detection pipeline for the Australian SKA Pathfinder VAST survey (2101.05898v2)

Abstract: The Australian Square Kilometre Array Pathfinder (ASKAP) collects images of the sky at radio wavelengths with an unprecedented field of view, combined with a high angular resolution and sub-millijansky sensitivities. The large quantity of data produced is used by the ASKAP Variables and Slow Transients (VAST) survey science project to study the dynamic radio sky. Efficient pipelines are vital in such research, where searches often form a `needle in a haystack' type of problem to solve. However, the existing pipelines developed among the radio-transient community are not suitable for the scale of ASKAP datasets. In this paper we provide a technical overview of the new "VAST Pipeline": a modern and scalable Python-based data pipeline for transient searches, using up-to-date dependencies and methods. The pipeline allows source association to be performed at scale using the Pandas DataFrame interface and the well-known Astropy crossmatch functions. The Dask Python framework is used to parallelise operations as well as scale them both vertically and horizontally, by means of a cluster of workers. A modern web interface for data exploration and querying has also been developed using the latest Django web framework combined with Bootstrap.