Data Provenance Initiative

• Data Provenance Initiative is a structured approach to tracking every data transformation and responsible agent, ensuring complete traceability of data products.
• It integrates automated capture in pipelines, such as CTA's ctapipe and SLURM-managed jobs, to log transformations and parameters in real time.
• The initiative employs standardized models like IVOA and W3C PROV along with scalable storage and API solutions to support efficient data lifecycle management and error tracing.

A Data Provenance Initiative is a structured and systematic approach to collecting, representing, and leveraging the full lineage of data products in large, complex, or distributed scientific environments. Provenance encompasses not only the chain of data transformations but also the specific agents (human or software), parameterizations, configurations, and execution environments that led to the current state of any given dataset. Initiatives in this domain are motivated by stringent requirements for traceability, reproducibility, quality assurance, and user trust in data products, especially in collaborative or publicly accessible research infrastructures (Servillat et al., 2018).

1. Provenance Requirements: Traceability and Reproducibility

In data-intensive science, provenance is not an optional metadata supplement; it is a core requirement to ensure trustworthiness and rigor of released products. Concrete requirements include:

• Traceability of every data product to its raw inputs, all calibration files, the specific software versions, execution parameters, and agents (human or software).
• Reproducibility such that any data product can be exactly regenerated by replaying the precise chain of activities, under identical conditions.
• Uniqueness and Persistence for identifiers representing Entities, Activities, and Agents.
• Comprehensive Parameterization, with all “used” and “generated” entities, timestamps, and parameter settings for each pipeline step being logged.
• Interoperable Formats to permit export and integration with external archives and services.

In the case of the Cherenkov Telescope Array (CTA), these requirements are encoded both in pipeline instrumentation (ctapipe, OPUS) and in the archival and API infrastructure, ensuring that provenance is inseparable from the data life cycle (Servillat et al., 2018).

2. Provenance Representation: The IVOA and W3C PROV Models

The standard approach to modeling provenance adopts directed acyclic graphs (DAGs) over three fundamental object classes:

• Entity (e.g., a data product, calibration file, intermediate result)
• Activity (a pipeline step, job, or operation)
• Agent (person, software module, or workflow controller)

Relationships are mapped using a formal schema:

$$\begin{aligned} &\texttt{Entity} \xrightarrow{\text{wasGeneratedBy}} \texttt{Activity}, \ &\texttt{Activity} \xrightarrow{\text{used}} \texttt{Entity}, \ &\texttt{Activity} \xrightarrow{\text{wasAssociatedWith}} \texttt{Agent}, \ &\texttt{Entity} \xrightarrow{\text{wasAttributedTo}} \texttt{Agent}, \ &\texttt{Entity} \xrightarrow{\text{wasDerivedFrom}} \texttt{Entity}. \end{aligned}$$

The IVOA Provenance model augments W3C PROV-DM with astronomy/physics-specific attributes, but the core structure remains the same: every data product is a node, with edges labeled “used” or “wasGeneratedBy” forming a detailed, formal provenance graph. This model allows arbitrary extension via additional attributes (e.g., software version, job ID, configuration) while preserving the graph’s formal semantics (Servillat et al., 2018).

3. Instrumentation and Automated Provenance Capture

Exemplary implementations directly integrate provenance capture into the computational pipeline. In the CTA’s ctapipe framework, every pipeline Task auto-instantiates provenance at start, logs all accesses to entities (inputs/outputs), and closes the lineage upon completion or failure—without requiring developers or users to write explicit provenance code. On distributed resources managed by SLURM, OPUS (Observatoire de Paris UWS System) extends this further, capturing job submissions, runtime events, agent identity, and collating all provenance dictionaries per job into a consolidated PROV-JSON or PROV-XML document (Servillat et al., 2018).

Automation is achieved by embedding provenance hooks in base pipeline classes and job schedulers, hiding the complexity from end-users and developers. This ensures not only completeness but also the consistency and granularity required for end-to-end traceability.

4. Storage, APIs, and Scalability

Provenance documents are serialized (PROV-JSON or PROV-XML) and stored co-located or linked with the scientific data products. A RESTful API layer is essential for enabling user and programmatic access to provenance:

• Retrieval: Fetch the full provenance graph for a given entity or job.
• Query: Filter or search by entity ID, agent identity, or attributes such as software version.
• Navigation: Traverse the “used” or “wasGeneratedBy” edges interactively to reconstruct the data product lineage.

Provenance capture occurs in-line during execution, with minimal runtime overhead—benchmarks on clusters with thousands of jobs per hour report additional load only in the few percent range, principally due to JSON serialization. Lightweight relational backends (SQLite, PostgreSQL) suffice, avoiding scaling bottlenecks for typical workloads in large astronomical facilities (Servillat et al., 2018).

5. Concrete Use Cases and Query Patterns

Concrete application scenarios highlight the breadth of provenance utility:

• Lineage Recovery: Given a derived science product, users retrieve and visualize the full transformation chain back to raw data, including intermediate parameters, software versions, and calibration steps. This allows precise reproducibility by resubmitting the same activity descriptions.
• Error Tracing: Systematic biases or anomalies in scientific output can be traced to suspect calibration files, incorrect agents, or environmental parameters. Timestamped and agent-tagged records enable rapid localization and invalidation of compromised calibration sequences.
• Provenance-Driven Reprocessing: In workflows where reprocessing is key (e.g., responding to improved calibration), provenance enables the selective rerunning of only affected branches of the processing graph, greatly improving computational efficiency.

6. Lessons Learned and Best Practices

Deployment across facilities such as CTA yields several best practices:

• Early Integration: Provenance capture must be integrated into pipeline frameworks at inception—not retrofitted—minimizing both technical debt and coverage gaps.
• Standardization: Adoption of community standards (IVOA PROV, W3C PROV-DM) ensures broad interoperability and eases downstream federation or cross-facility analysis.
• Automation and Abstraction: Provenance collection should be encapsulated in middleware or base classes, so all modules and processes inherit this functionality transparently.
• Scalable Job Management: Tying provenance event capture to job schedulers (e.g., through UWS or SLURM integration) ensures coverage of both workflow logic and execution environment, harmonizing pipeline and infrastructural provenance.
• User Tools and Visualization: REST APIs and graphical lineage explorers transform provenance from a compliance artifact into an accessible, actively used feature, empowering users to perform quality assurance, debugging, and compliance audits efficiently (Servillat et al., 2018).

7. Generalization and Blueprint for Large-Scale Facilities

The reference implementation—built on IVOA Provenance DM, ctapipe, and SLURM/OPUS—provides a blueprint for data-intensive projects seeking full lifecycle provenance. This model generalizes to any facility prioritizing traceability and reproducibility: every data product as an explicitly-identified node in a labeled, queryable provenance DAG, machine-actionable at all stages from raw ingestion through final publication, with robust APIs and automation at every layer (Servillat et al., 2018).