Task Curation Pipeline

• Task Curation Pipeline is an automated, rule-driven or learning-based system that selects, transforms, validates, and enriches data products for analytical tasks.
• It employs modular architectures with dependency graphs, curation tables, and decision rules to ensure scalability, reliability, and high throughput across diverse scientific domains.
• The system adapts to varying survey requirements and supports automated error recovery and minimal manual intervention, making it practical for vast astronomical and multimodal data archives.

A task curation pipeline is an automated, rule-driven or learning-based system for selecting, transforming, validating, and enriching data products specific to a target task or set of analytical requirements. Task curation pipelines span a wide spectrum of domains—from astronomical image archives to language, vision, and AI reward model curation—and share the objective of maximizing downstream utility, coverage, and efficiency through methodological rigor and automation. Below, the main architectural paradigms, decision rules, configurability strategies, instrument/survey adaptations, and scalability/performance characteristics are synthesized, primarily from the WFCAM/VISTA science archive pipeline (Cross et al., 2010), and contextualized by reference to broader developments.

1. Architectural Paradigms and Data Flow

Task curation pipelines adopt modular architectures structured as a dependency graph of tasks, each consuming and producing well-defined data products. In the WFCAM/VISTA Science Archives, the curation pipeline is organized into the following modules:

• Raw-data ingestion: Calibrated image files ("pawprints," tiles) and per-frame catalogs, along with auxiliary QC and reference catalogs, are ingested.
• Quality Control (QC): Automated scripts verify headers, exposure times, image quality metrics, and photometric zero-points; data failing QC are flagged and excluded.
• Programme setup: The "ProgrammeBuilder" generates a set of permanent "curation tables" from image metadata, defining all required products for the pipeline.
• Deep stacking and ingestion: Images are grouped and stacked along program-defined axes (e.g., by pointing, filter, and microstepping). In VISTA, pawprints are further coadded into tiles.
• Band-merging: Deep, band-merged source tables (across filters) are created per program.
• Recalibration: Single-epoch catalogs are recalibrated with respect to the deep reference catalog for astrometry and photometry.
• Per-epoch band-merged catalogs: Each recalibrated epoch is merged to yield multi-color, single-epoch source tables.
• Neighbour and cross-match tables: Internal neighbour tables link sources within a radius; cross-match tables connect archive sources to external catalogs (e.g., Gaia, SDSS).
• Synoptic tables: For positions/filters with ≥2 epochs, light curves and variability statistics are computed and injected into the archive.

This modular sequencing is orchestrated by DAG-style "what-next" logic, with each component triggered by the detection of missing products as determined by the curation tables (Cross et al., 2010).

2. Decision-Making Logic and Task Scheduling

Task progression in such pipelines is governed by set-difference logic on curated requirements vs. observed products. Concretely:

• Let $R$ be the set of required products (drawn from curation tables), $D$ the currently existing products, and $M = R \setminus D$ the missing products.
• If $|M|>0$, generate all $M$; otherwise, proceed to the next pipeline stage.
• After each task, data and history tables are updated, guaranteeing traceable, idempotent execution.

Two specialized decision rules are crucial in the WFCAM/VISTA system:

• Image grouping: Images are grouped for stacking by exact (header-driven) match on pointing center, filter, and microstepping, with instrument-specific fields (e.g. position angle, tile offsets for VISTA).
• Synoptic (multi-epoch) logic: For each pointing-filter, synoptic tables are built if and only if the epoch count $n_{\mathrm{epochs}}\ge 2$:

$$\mathrm{build\_synoptic} = \begin{cases} \mathrm{true}, & n_{\mathrm{epochs}} \ge 2 \ \mathrm{false}, & \text{otherwise} \end{cases}$$

All decision processes are implemented in SQL or Python, and the pipeline architecture is built on a repeatable, stateless comparison of "required" versus "current" products (Cross et al., 2010).

3. Instrument, Survey, and Task Adaptivity

Task curation pipelines support both instrument (hardware)-specific and survey (program)-specific logic:

• Common Module Set: Ingestion, QC, stacking, merging, recalibration, and table generation are generic across WFCAM and VISTA.
• VISTA-Specific Logic: Additional modules handle pawprint-to-tile merges, with tile-level metadata (position angles, offsets), and new tables (e.g., "RequiredTile," "ProductLinks").
• Survey Type Planning:
  • Shallow, wide-area surveys: Emphasize cross-matching, neighbour table construction for massive detection sets ($10^8$+ objects).
  • Ultra-deep, pencil-beam surveys: Prioritize iterative deep-stack refinement and advanced synoptic (light-curve/statistics) analysis.
• Automated Schema Generation: Program-specific SQL schemas, product definitions, and documentation are auto-generated from a single SQL-template + inline logic structure driven by the curation tables.

This flexibility allows the pipeline to process disparate scientific programs with varying depth, epoch count, and quality requirements without manual intervention, outside of limited QA flag overrides (Cross et al., 2010).

4. Automation, Configuration, and Failure Recovery

Curation pipeline configuration and execution are almost fully automated:

• Curation Tables as Control Interface: High-level requirements—such as cross-match catalogs, stacking depth, filters, and variability statistics—are encoded in tables like "RequiredStack," "RequiredTile," "RequiredFilters," and "RequiredNeighbours."
• Automatic Task Firing: Each block triggers solely on the detection of missing products. Human operators are not required for batch/PI programs; public surveys demand only minor manual specification for QC or exceptional situations.
• Resumable/Reentrant Execution: If a task fails (network interruption, error), logs and history tables precisely record missing products $M$, and the pipeline can be restarted from the failure point. This ensures high reliability with error/redo rates $<1\%$ over extended cycles.
• Scalable Task Distribution: New pointings ("width") and new epochs ("depth") are simply registered as additional rows in the relevant curation tables; the processing logic scales linearly with these dimensions.

The result is a robust, highly automated system where the only operator intervention required is for high-level survey configuration at program inception or periodic quality assurance review (Cross et al., 2010).

5. Scalability, Performance, and Operational Metrics

Task curation pipelines targeting petascale data volumes are architected for near-linear scalability and high throughput:

• Pipeline Throughput: $D$0–$D$1 detections/s end-to-end is attained on commodity 10-node compute clusters.
• Archive Scale: End-point databases comprise $D$2–$D$3 detections and $D$4 tables per survey, supporting deep/single-epoch/neighbour/synoptic products.
• Error/Redo Rate: Automatic task triggering achieves $D$5 manual intervention rate, with failed tasks picked up and retried in subsequent runs.
• Linear Scaling: New exposures (breadth) and epochs (depth) are handled additively; code paths for stacking, band-merging, and recalibration run independently per logical grouping.
• Full Re-processing Cadence: Large surveys (hundreds of deg$D$6) can be fully reprocessed—raw data to band-merged, synoptic-ready archives—on a week timescale, bounded principally by I/O and QC bottlenecks rather than pipeline logic.

This model has proven practical not only for infrared/astronomical curation but establishes principles—separation of configuration and task logic, declarative requirements tracking, error-resilient automation—that are being generalized across scientific data infrastructures (Cross et al., 2010).

6. Broader Methodological Context and Comparative Pipelines

While the WFAU pipeline is anchored in astronomy, structurally analogous curation architectures are prominent in other technical domains:

• Hybrid human-AI curation (e.g., Skywork-Reward-V2): Initial human-verified "seed" labels and protocol-driven annotation guide LLM-based auto-label expansion, leveraging controlled iterative consistency and quality gates for scaling to tens of millions of examples (Liu et al., 2 Jul 2025).
• Reinforcement-learned curation (SkillOS): An explicit RL skill curator maintains an evolving repository of structured content by maximizing group-wise long-horizon utility, quality, and conciseness via grouped reward optimization (Ouyang et al., 7 May 2026).
• Modular LLM-driven task compilers (SEED): Dynamic assembly and optimization of hybrid module pipelines (CodeGen, CacheReuse, ModelGen, LLM) with cost-accuracy skylining optimize execution for arbitrary domain-specific curation (Chen et al., 2023).
• Multimodal/vision-language synthetic curation (MM-Gen): Structured partitioning (CLIP-based), in-context exemplar-driven text generation, and test-of-perplexity filtering for VLM fine-tuning (Joshi et al., 7 Jan 2025).

All state-of-the-art curation pipelines emphasize the same core principles: declarative encoding of requirements; idempotent, automatable task triggering; unified schema and product specification; robust recovery and scalability; and, increasingly, adaptive integration of learning systems for quality, efficiency, and generalization across domains.

---

References:

(Cross et al., 2010, Liu et al., 2 Jul 2025, Ouyang et al., 7 May 2026, Chen et al., 2023, Joshi et al., 7 Jan 2025)