Data Provenance Audit Overview

• Data provenance audit is a framework that systematically tracks data origins, transformations, and usage to ensure accountability and integrity.
• It employs diverse capture strategies, such as whole-system logging, log-based extraction, and language-integrated provenance, to provide traceability.
• The audit process supports compliance and governance across various domains by enabling transparent, secure, and reproducible data workflows.

Data provenance audit concerns the systematic use of provenance—information about the origin or derivation of data—to inspect how data, workflows, and related artifacts were created, transformed, shared, and consumed. In the research literature, provenance audit is tied to accountability, reproducibility, compliance, forensics, and integrity verification, and it appears in forms ranging from field-level “where-provenance” and row-level lineage to cross-organizational decision pipelines and long-lived version histories (Fehrenbach et al., 2016, McClatchey et al., 2015, Singh et al., 2018).

1. Core concepts and provenance semantics

At its most general, provenance is represented as relationships among entities, activities, and agents. In whole-system capture, entities include kernel data objects such as files, messages, packets, xattrs, inode attributes, exec parameters, and network addresses; activities are typically processes operating on entities; and agents are users, groups, persons, or organisations on whose behalf activities run. This graph structure is used to express both causal dependencies and information-flow relationships, and it is directly aligned with the W3C PROV Data Model (Pasquier et al., 2017).

Two audit semantics recur throughout the literature. Where-provenance records where a result value was “copied from” in the input, typically as a source table, field, and row identifier; lineage records the set of input database records accessed in producing an output record, and thereby answers why that output is present (Fehrenbach et al., 2016). The distinction is consequential: where-provenance is suited to correcting a wrong value in a copied field, whereas lineage is suited to reconstructing the input records that justified a derived row.

The audit scope is often broader than value lineage alone. In big-data medical systems, provenance audit is described as requiring both data lineage and process lineage, because datasets, workflow specifications, parameters, errors, outputs, and workflow definitions all evolve over time. The problem is therefore not merely logging static lineage, but preserving changing workflows, changing datasets, changing definitions, and changing use patterns across a long-lived research environment (McClatchey et al., 2015).

A further extension is decision provenance, which applies provenance methods to “decision pipelines”: chains of inputs to, the nature of, and the flow-on effects from, the decisions and actions taken at design-time and run-time throughout interconnected systems. This widens provenance audit from data artifacts alone to systems-of-systems, including models, rule engines, transfers, downstream actions, and responsible organisations (Singh et al., 2018).

2. Audit objectives and evidentiary functions

The literature treats provenance audit as an evidentiary substrate for several distinct but overlapping purposes. In large-scale medical research infrastructure, provenance is required simultaneously for accountability—“who did what, to which data, when, and for what purpose”—for reproducibility and validation of results, for compliance-oriented traceability, for collaborative reuse, and for retrospective analysis such as error detection and investigation of exceptional behavior (McClatchey et al., 2015).

In digital forensics, the provenance record is described as the evidentiary backbone of chain-of-custody. The required properties extend beyond trace logging to include integrity, traceability, accountability, reproducibility, secure authentication and access control, immutability, fast provenance extraction with verification, version tracking, and secure communication among components. In that setting, an audit must establish not only what happened to evidence, but whether the retrieved history itself is authentic and untampered (Akbarfam et al., 2023).

Several works emphasize that provenance itself can become the object of audit. The clinical decision support literature therefore introduces meta-provenance—a provenance trace of the provenance-construction service itself—and a complementary non-repudiation layer that generates evidence for each service call, signs it, annotates the service trace, and submits a representation of that evidence to a provider-agnostic notary service. One of its strongest claims is that “a meta-provenance record is sufficient to reconstruct an object-level provenance document in its entirety” (Fairweather et al., 2020).

This evidentiary framing also explains why provenance audit is central to governance. Decision provenance is explicitly presented as a mechanism for oversight, audit, compliance, risk mitigation, investigation after failures, and user empowerment in algorithmic systems, especially where information flows cross technical and organisational boundaries (Singh et al., 2018).

3. Capture and representation strategies

A major design axis is where provenance is captured. In workflow-centric systems, provenance is integrated into the execution model itself. The neuGRID/N4U architecture uses CRISTAL, a “description-driven” workflow and provenance tracking system in which objects are modeled as Items with lifecycles, descriptions, instances, analyses, datasets, pipelines, algorithms, analysis elements, parameters, results, users, and annotations. It records workflow specifications, data or inputs supplied to each workflow component, annotations, links and dependencies between workflow components, execution errors, outputs, exact start and completion times, grid resources used, and final result locations (McClatchey et al., 2015).

A second strategy is whole-system capture. CamFlow implements this in Linux through the Linux Security Module framework, NetFilter hooks for outgoing packets, and relayfs for exporting events to user space. It captures provenance at the operating-system boundary, models packets as entities, extends provenance across machines, and uses the concept of a state-version of kernel objects to guarantee that the provenance graph remains acyclic (Pasquier et al., 2017).

A third strategy is application- and service-level interception. DAP retrofits web services by interposing on application components’ communications rather than instrumenting application code or modifying the DBMS. It uses minimal knowledge of open protocols, captures web request boundaries in Apache, proxies application-to-database traffic, parses SQL, and emits provenance tuples of the form <pid, relationship, column, table>, thereby linking request-level, database-level, and host-level provenance in a unified graph (Bates et al., 2016).

A fourth strategy is log-based extraction. OneProvenance reconstructs dynamic coarse-grained provenance from query event logs by identifying query execution dependencies, building activity-specific QQTrees, extracting runtime metadata, and rolling statement-level lineage up to query-run and static-query levels. Its QQTree construction is explicitly stated to run in $\mathcal{O}(n)$ time for $n$ activity events and to use $\mathcal{O}(m)$ space where $m$ is the number of started events (Psallidas et al., 2022).

A fifth strategy is language-integrated provenance. In Links, both where-provenance and lineage are compiled by rewriting and normalizing monadic comprehensions, while the type system distinguishes provenance metadata from normal data. The implementation requires no changes to the underlying database system (Fehrenbach et al., 2016).

These strategies are not mutually exclusive. Decision provenance explicitly distinguishes disclosed provenance, built into application code or APIs, from observed provenance, captured by observing behavior externally; it argues that exposing full decision pipelines will likely require several complementary capture regimes across different layers (Singh et al., 2018).

4. Integrity, immutability, and authorization mechanisms

Provenance audit systems differ most sharply in how they make the audit trail itself trustworthy.

System	Audit setting	Integrity and control mechanism
CRISTAL / neuGRID-N4U	Big-data medical workflows	Preserves historical versions of Items, definitions, and application logic, together with who made the change, when, and for what purpose (McClatchey et al., 2015)
ForensiBlock	Digital forensics	Private blockchain, RBAC-SA, off-chain storage retrieval with Merkle root verification, and a distributed per-case Merkle root (Akbarfam et al., 2023)
SciChain	HPC scientific provenance	Permissioned replicated ledger, POST consensus with PSS and POET, in-memory compute-node blockchains plus shared-storage “ground truth,” under a “more than 50%” validating threshold (Al-Mamun et al., 2020)
DataProv	Scientific document provenance	Ethereum smart contracts, encrypted hashes, OPM-based event records, and stakeholder voting before a provenance change is admitted (Ramachandran et al., 2017)
Scrybe	Clinical trial data fusion	Permissioned blockchain storing proof of provenance for an off-chain changelog, with signatures, hashes, timestamps, and previous-block linkage (Oakley et al., 2021)
Federated Hyperledger Fabric architecture	Big-data science in federated settings	PID-based traceability, checksums, provenance versioning, ACLs, endorsement policies, and PKI-backed client identities over Hyperledger Fabric (Marchioro et al., 30 May 2025)

A common pattern is the separation of off-chain payloads from on-chain integrity anchors. ForensiBlock keeps encrypted forensic data and organized provenance records off-chain while storing case metadata, access-control state, tokens, and distributed Merkle roots on-chain (Akbarfam et al., 2023). Scrybe stores detailed changelog entries off-chain and only their hashes and non-identifying metadata on-chain, explicitly to preserve HIPAA compliance (Oakley et al., 2021). The federated big-data-science architecture stores artifacts and provenance documents off-chain while placing checksums, URIs, versions, and timestamps on a permissioned blockchain (Marchioro et al., 30 May 2025).

A parallel line of work strengthens provenance audit through non-repudiation rather than blockchain. In clinical decision support, each provenance-service action is bound to a signed token with a trusted timestamp, a token identifier, certificates, and a notarized commitment; only a signed hash of the signed token is sent to the notary, so sensitive provenance need not leave the local system (Fairweather et al., 2020).

5. Querying, reproduction, and proactive enforcement

Provenance audit is not limited to storage; it depends on usable query and replay mechanisms. In neuGRID/N4U, the Querying Service and Analysis Base let users browse past analyses and their results, retrieve specific workflow versions, examine the results of each individual computation, clone complete analyses, alter parameters or selected subjects, and rerun them (McClatchey et al., 2015). OneProvenance provides the complementary database view: drill-down to statement-level provenance and roll-up to higher-level query and procedure provenance, while preserving user identity, timestamps, client application, server, CPU time, duration, and row counts (Psallidas et al., 2022).

Several systems make provenance operational at run time. CamFlow streams provenance to tenant-built auditor applications and illustrates three such uses: continuous compliance monitoring, fault/intrusion detection through FRAPpuccino, and data loss prevention through graph queries or taint propagation (Pasquier et al., 2017). DAP uses fast ancestry queries to power a provenance-aware Network Guard that inspects outbound web responses against rules of the form < RULE, [ Tab_1.Col_1, \dots, Tab_n.Col_n], SIZE> before release (Bates et al., 2016).

ACDC goes further by defining provenance policies as predicates over provenance subgraphs that are evaluated before a capsule is used as input to a function. In that model, provenance becomes a precondition for admissibility: data whose lineage violates a contract-specified predicate can be blocked before it is consumed (Engram et al., 2021).

The phrase “data provenance audit” also appears in a narrower machine-learning sense: auditing whether a datum was used to train a model. DPDA addresses that problem by comparing model outputs on original and transformed inputs, and it explicitly notes that point-level auditing is a special case with $|D_i|=1$ (Mu et al., 2022).

6. Domains, misconceptions, and open issues

Data provenance audit has been applied in Alzheimer’s biomarker research, digital forensics, high-performance scientific computing, web-service intrusion tracing, clinical decision support, clinical trials, sport performance analysis, federated scientific infrastructures, and large-scale AI dataset licensing and attribution (McClatchey et al., 2015, Akbarfam et al., 2023, Al-Mamun et al., 2020, Oakley et al., 2021, Simmons et al., 2018, Marchioro et al., 30 May 2025, Longpre et al., 2023).

A persistent misconception is that provenance automatically guarantees truth. Multiple systems explicitly separate integrity of the provenance record from truth of the recorded content. Scrybe is strongest on this point: it makes recorded history tamper-evident, but if an authorized researcher records false data through the proper workflow, the system preserves that false history faithfully (Oakley et al., 2021). The non-repudiation literature similarly stresses that signatures alone are insufficient without trusted timestamps, evidence retention, storage assumptions, and adjudication-oriented design (Fairweather et al., 2020).

A second misconception is that more capture always means complete auditability. CamFlow states plainly that tailored provenance “is no longer guaranteed to be complete,” and OneProvenance is explicit that coarse-grained provenance does not answer row-level or cell-level influence questions (Pasquier et al., 2017, Psallidas et al., 2022). The sport-provenance literature generalizes the point: one-size-fits-all provenance and workflow systems are “a poor fit in practice,” especially when workflows mix manual and automated activities or require domain-specific notation and causality models (Simmons et al., 2018).

A third recurring issue is governance. In big-data medical systems, dynamic modification of running workflows is powerful, but the same paper notes that security, access control, and governance are only lightly addressed, and that live process mutation can raise governance questions in regulated settings (McClatchey et al., 2015). Decision provenance adds that provenance records may themselves be highly sensitive and that raw provenance is often too complex for stakeholders to use directly, requiring audience-specific views and secure disclosure controls (Singh et al., 2018).

Finally, provenance audit now extends beyond runtime traceability to the legal and attributional history of datasets themselves. A large-scale audit of 1858 individual text datasets found license omission rates of 72% on GitHub, 69% on Hugging Face, and 70% on Papers with Code, while the abstract summarizes the broader pattern as “license omission of 70%+ and error rates of 50%+.” That result situates provenance audit squarely within AI data governance: not only as a record of transformations and executions, but as a trace of creators, sources, series of license conditions, and subsequent use (Longpre et al., 2023).