Automated curation of infra-red imaging data in the WFCAM and VISTA Science Archives (1012.4107v1)

Abstract: The two fastest near infrared survey telescopes are UKIRT-WFCAM and VISTA. The data from both these instruments are being archived by Wide Field Astronomy Unit (WFAU) at the IfA, Edinburgh, using the same curation pipeline, with some instrument specific processing. The final catalogues from these surveys will contain many tens of billions of detections. Data are taken for a range of large surveys and smaller PI programmes. The surveys vary from shallow hemisphere surveys to ultra deep single pointings with hundreds of individual epochs, each with a wide range of scientific goals, leading to a wide range of products and database tables being created. Processing of the main surveys must allow for the inclusion of specific high-level requirements from the survey teams, but automation reduces the amount of work by archive operators allowing a higher curation efficiency. The decision making processes which drive the curation pipeline are a crucial element for efficient archiving. This paper describes the main issues involved in automating the pipeline.