Automated Curation Pipeline

• Automated Curation Pipeline is a modular, task-driven framework that ingests, verifies, and integrates high-throughput infra-red imaging data into analysis-ready databases.
• It leverages a sequential process including quality control, metadata-driven grouping, and automated decision logic to ensure scalability, traceability, and consistent data products.
• Advanced SQL templating and instrument-specific adaptations, combined with log-based state management, deliver robust, extensible, and recovery-ready processing.

An automated curation pipeline is a modular, task-driven data processing framework designed to fully automate the ingestion, verification, integration, and management of high-throughput scientific data, with the goal of producing high-quality, analysis-ready, and scientifically robust databases. In the context of large-scale infra-red imaging surveys as exemplified by the WFCAM and VISTA Science Archives, such a pipeline orchestrates instrument-specific workflows, automates decision-making to trigger curation steps, and ensures both scalability and traceability as survey data volumes reach tens of billions of detections.

1. Modular Pipeline Structure

The pipeline architecture is organized into a tightly coupled sequence of functionally distinct tasks initiated upon receipt of new, calibrated infra-red imaging data—typically supplied by preliminary reduction centers (e.g., CASU). These sequential tasks are:

• Quality Control of ingested data
• Automated Programme Setup via metadata-driven grouping (notably the ProgrammeBuilder class)
• Deep product and catalogue creation and ingestion
• Band-merging to construct a master Source table from the deepest stacked images
• Per-epoch recalibration
• Epoch-wise band-merged catalogue creation
• Generation of neighbor tables for cross-match operations (internal and external)
• Synthesis of synoptic tables required for light-curve and variability analysis

The pipeline is visually represented as a directed acyclic task graph, wherein yellow rectangular nodes correspond to processing steps and diamond nodes denote control structures for product verification.

A schematic of the ProgrammeBuilder logic demonstrates metadata grouping by observational parameters—position, filter, microstepping for both UKIRT-WFCAM and VISTA, and for VISTA, further partitioning by position angle and tile/grid offsets. This grouping informs the construction of control tables (RequiredStack, RequiredTile, etc.), which in turn drives the instantiation of survey-specific schema via a flexible, parameterized SQL template system.

2. Automation Mechanics and Decision Logic

Automation is central to minimizing manual curation workload and increasing overall throughput and reliability. Automation mechanisms in the pipeline include:

• Systematic “expected-versus-actual” product verification, wherein control tables (e.g., RequiredStack, RequiredTile) define the set of required products and the pipeline continually cross-references these with actual entries in data tables (e.g., Multiframe, ProgrammeFrame)
• Autonomous triggering of downstream tasks: if a required product is incomplete, missing, or inconsistent, the relevant processing task is automatically queued and executed; completion of prerequisite tasks triggers subsequent pipeline stages, forming a robust workflow loop
• Control structures in schema generation: an advanced SQL templating syntax (“++c:a” and “==c:a”) is utilized to automatically loop over available filters and color combinations per survey, instantiating database tables and columns as required—removing the need for hand-edited schema variations
• State management and failure recovery: each stage writes logs and updates dedicated curation history tables, enabling detection of failed/incomplete tasks and safe automated resumption or manual intervention

A core “decision formula” operationalizes task execution:

$$\text{If } \text{Expected Products from Curation Tables} \neq \text{Actual Products in Data Tables}: \quad \text{Trigger creation of missing products;} \qquad \text{Else: continue.}$$

This logic permeates the pipeline, safeguarding consistency, completeness, and resilience to interruptions.

3. Instrument-Specific Adaptation

While the pipeline’s core logic is unified across UKIRT-WFCAM and VISTA, instrument heterogeneity requires context-sensitive adaptation:

• Input image metadata is initially processed uniformly, but VISTA’s tile-based survey design necessitates additional steps: pawprints are grouped into tiles (captured in RequiredTile); ProductLinks tables map raw pawprint stacks to assembled tiles, factoring in VISTA-specific multiplicity (e.g., position angle, offset)
• Adaptable SQL schema templates with substitution strings and control logic enable the pipeline to produce instrument- and programme-specific tables and columns, incorporating filter names, photometric systematics, and meta-information as needed
• ProgrammeBuilder and curation steps incorporate logic for instrument-differentiated processing without needing divergent code bases

This approach allows a single automated curation pipeline to accommodate diverse survey and instrument requirements with minimal manual customization.

4. Data Management and Schema Generation

Data management within the pipeline is characterized by the use of dynamic control tables and on-the-fly schema generation:

• Control tables (e.g., RequiredStack, RequiredTile, RequiredNeighbours) formally encode the specifications for products—a survey's processing plan is thus captured in relational metadata
• Schema generation employs an SQL template language with embedded loops and substitution—e.g.,

++c:a **s*&a&m&b&Pnt real not null, --/D Point source colour &As&-&Bs& (using aperMag3) --/U mag --/C PHOT_COLOR --/Q &a&AperMag3,&b&AperMag3 --/N -0.9999995e9 --/G allSource::colours ==c:a

which systematically produces color columns for each filter combination.

• The products themselves—deep stacks, merged source lists, neighbor and synoptic tables—are highly volumetric; database physical design is thus tailored for scalable access to tens of billions of detections, accommodating both survey-wide and PI-driven workflows

Logs and curation histories are systematically recorded, tracking all metainformation for full auditability and provenance.

5. Scientific and Operational Flexibility

Automated curation must support the broad heterogeneity of surveys, from shallow, wide-area hemisphere scans to deep, narrow-field time-domain campaigns:

• Survey-specific high-level requirements (e.g., bespoke quality control scripts, custom photometric calibrations, or additional cross-matching) can be encoded at the configuration/programme setup level
• Flexible processing steps—parameterized by survey metadata—allow optimization for depth, cadence, and product type; operational throughput is tunably coupled to the complexity and volume of the input campaign
• Synoptic tables and neighbor tables are constructed for scientific analyses such as variability studies and cross-survey correlation, reflecting the pipeline’s suitability for a wide range of astrophysical investigations

The pipeline design explicitly supports “scaling up to many tens of billions of detections,” ensuring continued relevance and utility as data rates increase.

6. Robustness, Extensibility, and Database Consistency

The pipeline’s robustness is ensured by:

• Safe task replay: built-in crash recovery and resumption mechanisms, log-based and table-based state management, and “expected-versus-actual” checks prevent duplication, omission, and data corruption during processing
• Extensibility and maintainability: advanced schema templating minimizes hand-coded special cases, enabling rapid adaptation to new instruments, processing demands, or data organization paradigms
• Consistency guarantees: by integrating product verification into each control structure, the pipeline enforces end-to-end consistency, maintaining strict alignment between expected scientific outputs and the realized database state

Potential system failures due to network or software errors are mitigated by the automated, log-driven restart strategy.

7. Challenges and Solutions

Principal challenges and their corresponding mitigations include:

• Data heterogeneity: differing demands for wide, sparse, and deep, intensive surveys are accommodated by parameterizing procedures and time allocation by survey metadata such as epoch count and target depth
• Instrumental differences: discrepancies (e.g., VISTA’s tile grouping and offset handling) are integrated through programmable groupings and link tables
• Automation resilience: interruptions and discrepancies in data flow are systematically absorbed by verifying data product existence at each step and enabling reliable recovery
• Database schema consistency: control structures and substitution logic ensure minimal code duplication and ease future schema transition or extension

Conclusion

The automated curation pipeline instantiated for WFCAM and VISTA Science Archives embodies a scalable, modular, and highly automated framework for processing and organizing large-scale infra-red imaging data. Robust “expected-versus-actual” verification, control table-driven orchestration, flexible instrument adaptation, resilient task automation, and detailed state tracking characterize the pipeline’s architecture. This design delivers high curation efficiency, database consistency, and scientific flexibility—enabling the astronomy community to access, analyze, and exploit data volumes at the order of tens of billions of detections without being stymied by manual curation bottlenecks (Cross et al., 2010).