Data Lineage vs Data Provenance: flow tracking versus origin evidence for data and AI

Teams use "lineage" and "provenance" as if they were the same thing, and then discover during an audit, a poisoning investigation or an AI Act review that the evidence they kept answers only half the question. Lineage records how data flows: which jobs read which inputs and wrote which outputs, ideally down to the column. Provenance records where data came from and whether that origin can be trusted: who issued it, under what licence or consent, how it was generated, and whether it is still the same bytes that were approved. This guide compares the two for enterprise data and AI pipelines: the threat each addresses, how each is built, where each fails and how to combine them into evidence an assessor will accept. It is for data platform owners, security architects and AI governance leads; lineage mechanics are covered in the Data lineage, data quality and data contracts as security controls guide.

Key takeaways

• Lineage answers "where did it go and what touched it", provenance answers "where did it start, who vouches for it and has it changed". You need lineage to scope an incident and provenance to decide whether to trust the data at all.
• Lineage is usually observed, provenance is usually asserted. Pipelines emit lineage as a side effect of running; provenance has to be captured at the point of acquisition and protected with hashes or signatures, or it is just a label anyone can edit.
• Neither stops data poisoning on its own. Provenance tells you which sources to trust and lets you detect substituted content; lineage tells you which models, features and indexes a bad source reached. Poisoning response needs both.
• The EU AI Act asks for both in different words. Article 10(2)(b) asks for data collection processes and the origin of data; Article 10(2)(c) asks for preparation operations such as annotation, labelling, cleaning, enrichment and aggregation. The first is provenance, the second is lineage.
• Keep your own evidence store. Platform lineage retention windows and editable catalogue tags are too weak for audit; export lineage events and seal provenance records with content hashes so both survive the look-back period.

The short answer

Data lineage is the recorded graph of inputs, transformations and outputs: dataset A and dataset B were read by job J at run R, which wrote dataset C, and column C.email was derived directly from A.email without masking. It is a property of your pipelines, it changes every time a job runs and it is best captured automatically by the scheduler, engine or warehouse.

Data provenance is the record of origin and custody for a piece of data or a dataset: who created or issued it, when and how it was generated, under what licence, consent or contract it may be used, and a fingerprint that proves the copy you hold is the one that was described. The W3C PROV family models this as entities (the data), activities (what produced it) and agents (who is responsible), linked by relations such as wasGeneratedBy, used, wasDerivedFrom and wasAttributedTo.

The two overlap: PROV's "used" and "wasGeneratedBy" relations can express lineage. In practice lineage is for impact and flow, provenance for trust and accountability. A dataset can have perfect lineage inside your estate and no provenance at all, which is exactly the position of most web-scraped or vendor-supplied training data.

Side by side

Data lineage in depth: threat model, architecture and failure modes

Threat model

Lineage defends against loss of control over data you already hold: sensitive columns that lose classification or masking as they are copied, unexpected flows into exports or AI stores, and not knowing which reports, features or models consumed a dataset that proved wrong, poisoned or unlawfully collected. Without it, incident scoping becomes interviews and deletion requests cannot be honoured completely.

Architecture in words

Runtime emitters sit in the orchestrator and compute engines and send events to a lineage backend. In the OpenLineage model, a RunEvent records a job run's state changes (normally START, then COMPLETE or FAIL), with input and output datasets attached; jobs are named within a namespace, runs carry a client-generated UUID, and datasets are named from their physical location. Design-time JobEvents and DatasetEvents add static metadata such as source code location, schema and ownership. Warehouses add their own native lineage or access history. A catalogue stitches these into a graph and links it to classification tags.

Failure modes

• Uninstrumented paths: notebooks, ad hoc scripts, SaaS exports and copy jobs outside the orchestrator create edges nobody records.
• Coarse granularity: table-level lineage cannot show whether a restricted column survived a transformation.
• Retention shorter than the look-back: platform lineage that ages out before an audit or a slow-burn investigation is evidence you no longer have.
• Editable edges: manual lineage that anyone with catalogue rights can add or remove can hide a flow.

Controls

Instrument the orchestrator and main engines first, require column-level lineage for restricted data, export events to an append-only store you control, and treat manual lineage edits as changes to a security record.

Data provenance in depth: threat model, architecture and failure modes

Threat model

Provenance defends against trusting data you should not. The threats are poisoned or manipulated external data, content substituted between the time it was vetted and the time it was used, datasets collected without a lawful basis or under a licence that forbids your use, and synthetic or edited media presented as original. MITRE ATLAS lists Publish Poisoned Datasets (AML.T0019) and Poison Training Data (AML.T0020) as techniques, and its mitigation Maintain AI Dataset Provenance (AML.M0025) asks for a detailed history of each dataset's source and a complete record of modifications.

Architecture in words

Provenance is captured at the boundary where data enters your control. For each acquired dataset, record the fields the Data and Trust Alliance standard groups as source (title, issuer, unique identifier, metadata location, description), provenance (original source if different from the issuer, origin geography, issue date, previous version date, date range of generation, generation method, format) and use (confidentiality classification, consent documentation location, privacy-enhancing techniques applied, processing and storage geography, licence, intended use, proprietary data presence). Bind that record to the content with a cryptographic hash of every file or shard, and where the supplier can sign, verify the signature. In W3C PROV terms the dataset is an entity attributed to an agent (the issuer), generated by an activity (collection, synthesis, labelling), and a bundle of provenance assertions can itself carry provenance, which is how you record who wrote the provenance record.

For images, audio and video, C2PA Content Credentials do the binding for you: a signed manifest of assertions about origin, edits and AI use, tied to the asset with a cryptographic hash (a hard binding) so any change breaks the match, with ingredients forming a provenance tree. C2PA is explicit that it does not judge whether content is true; it confirms the provenance data is well formed, unaltered and signed by a trusted signer. See the AI Content Provenance and C2PA Guide.

Failure modes

• Metadata without binding: a licence field and a source URL with no hash prove nothing about the bytes you trained on.
• Mutable sources: data referenced by URL can change after it was described, so the content fetched at training time differs from the content that was vetted.
• Provenance stripped in transit: re-encoding drops embedded manifests, and pipeline copies lose the dataset record unless the catalogue carries it forward.
• Unverified assertions: a supplier's statement of consent or licence that nobody checked is recorded with the same weight as one that was.

Controls

Gate acquisition on a provenance record, hash every artefact at ingestion, pin jobs to those hashes rather than paths, verify supplier signatures, store records separately from the data with versioning, and give each external source an owner. The AI Training Data Provenance and Copyright Exposure Guide covers the licence and copyright side in depth.

What each one contributes to poisoning defence

OWASP's LLM04:2025 Data and Model Poisoning describes poisoning across pre-training, fine-tuning, embedding and transfer learning, names split-view and frontrunning poisoning as examples, and lists tracking data origins and transformations, vetting data vendors, data version control and anomaly detection among its mitigations. Split-view poisoning is the clearest illustration of why provenance needs a content hash: the data at a referenced location is not the data that was reviewed. Use the table to see which control answers which stage.

Lineage alone shows how far a problem spread but not that it exists; provenance alone shows a source is bad but not what to roll back. See the AI Data Poisoning and Backdoors Guide.

Regulatory evidence: what the AI Act and privacy law ask for

• EU AI Act Article 10(2): training, validation and testing data for high-risk systems must be subject to data governance practices covering, among other things, data collection processes and the origin of data (and for personal data the original purpose of collection) in point (b), and data preparation operations such as annotation, labelling, cleaning, updating, enrichment and aggregation in point (c). Point (b) is a provenance requirement and point (c) is a lineage requirement. Application dates for high-risk obligations were moved by the 2026 Digital Omnibus amendments, with Annex III systems reported as applying from 2 December 2027; check the current consolidated text before planning. See the EU AI Act Article 10 Data and Data Governance Guide.
• GDPR and UK GDPR: records of processing and DPIAs need personal data flows (lineage); lawful basis and purpose limitation need to know how it was collected (provenance). See the Data Mapping and ROPA Guide.

Tooling and evidence

Software build provenance (SLSA, in-toto attestations) answers the same origin question for code and artefacts rather than data; see the Provenance Attestation vs SBOM comparison and the AI Supply Chain and AI-BOM Guide for the model and dataset bill of materials.

Worked scenario: a fine-tuning set from three sources

A bank fine-tunes a customer service model on three inputs: an internal ticket export, a licensed third-party financial glossary and a public question-and-answer corpus referenced by URL. Six weeks after release, the model starts recommending a specific external website for password resets.

• With lineage only: the team sees which three datasets the run read and which two applications use the model, but not which source introduced the behaviour or whether the corpus changed after review.
• With provenance only: re-hashing the training copy against the March review manifest shows 214 files differ, but the team must search manually for every other dataset, index and model that consumed the corpus.
• With both: the hash mismatch identifies the substituted content, the lineage graph shows the same corpus also fed a retrieval index used by an internal assistant, and the run records identify the exact model versions to withdraw. Remediation is a scoped rollback to the last model trained on verified hashes, rebuilding the index, and adding a gate that blocks any training job whose input hashes do not match an approved provenance record.

Decision criteria

• Start with lineage when: most of your risk is internal data movement, you have a central warehouse or lakehouse with native lineage, and your immediate drivers are privacy records, deletion requests and incident scoping.
• Start with provenance when: you train, fine-tune or ground models on external or purchased data, you publish or ingest media where authenticity matters, or you must prove licence and consent for data you did not create.
• Do both when: you operate high-risk AI systems, run retrieval over mixed internal and external content, or have had a data integrity incident.
• Small versus large: small teams can use native warehouse lineage and an acquisition register with a hash per file in version control; large estates should standardise on an open lineage format, sign high-value datasets and join both in the catalogue.

How to implement the combined model

1. Inventory sources and sinks: list external data sources and AI pipelines, marking mutable sources (URLs, APIs, shared buckets).
2. Define a minimum provenance record: adopt the Data and Trust Alliance fields as a starting schema, add a content hash per artefact and an accountable owner, and make the record mandatory before data lands in a governed zone.
3. Turn on lineage capture: enable platform-native lineage, add OpenLineage emitters to the orchestrator and main engines, and require column-level lineage for restricted and personal data.
4. Pin runs to hashes: change training and evaluation jobs to reference dataset versions and hashes, not paths. Pilot on one pipeline in report-only mode that logs mismatches before blocking.
5. Export and seal evidence: copy lineage events and provenance records to an append-only store with retention matched to your longest audit or investigation look-back.
6. Enforce and roll back safely: switch the hash gate from report-only to blocking once false mismatches are resolved; keep a documented override that records who approved an exception, and revert to report-only if a supplier change breaks legitimate loads.

Where identity fits

Both records name agents: the service account or workload identity that wrote an output, and the key that signed a dataset or manifest. If those identities are shared or unowned, anyone using them could have written the edge or signed the record. Give pipeline and signing identities single owners, scope them to the datasets they need and rotate their credentials. See the NHI Discovery Across Cloud, SaaS, CI/CD and On-Premises Environments guide.

Detection and measurement

• Provenance coverage: percentage of external datasets used in AI pipelines with a complete provenance record and content hash.
• Hash pinning rate: percentage of training and evaluation runs that reference pinned dataset versions rather than paths.
• Column lineage coverage: percentage of restricted columns with downstream column-level lineage.
• Hash mismatch events: count and time to resolution; any unexplained mismatch on an approved source is an integrity incident.
• Time to impact list: time from identifying a bad source to a complete list of affected models, indexes and reports.

Framework mapping

Common failure patterns

• Calling a catalogue tag "provenance": a free-text source field with no hash or signature cannot detect substitution and will not satisfy an assessor.
• Training on URLs, not hashes: referencing mutable locations makes split-view style substitution invisible to every downstream control.
• Lineage that stops at the warehouse: notebooks, feature stores and vector indexes, where AI pipelines diverge, are left out.
• Relying on platform retention: lineage that ages out after weeks or a year disappears before the investigation or audit that needs it.
• Shared pipeline identities: when every job runs as one service account, lineage cannot say who changed what and provenance signatures cannot be attributed.

Practitioner checklist

• Require a provenance record with issuer, licence, consent location, generation method and a content hash before external data enters a governed zone.
• Pin training, evaluation and index builds to dataset versions and hashes; log mismatches before blocking.
• Enable column-level lineage for restricted and personal data and close uninstrumented paths found in access history.
• Export lineage events and provenance records to an append-only store with retention matched to audit look-backs.
• Link provenance records to lineage nodes in the catalogue so origin and impact can be queried together.
• Give every pipeline and signing identity a named owner and scoped permissions.
• Map evidence to AI Act Article 10(2)(b) and (c) for high-risk systems and rehearse an impact walk from a single bad source.

Standards and references

• W3C: PROV-Overview, An Overview of the PROV Family of Documents
• W3C: PROV Model Primer
• OpenLineage: Object Model
• EU AI Act: Article 10, Data and Data Governance
• Data and Trust Alliance: Data Provenance Standards
• C2PA: Technical Specification Explainer
• OWASP: Top 10 for LLM Applications 2025 (LLM04:2025 Data and Model Poisoning)
• NIST: AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations

Related NHI Mgmt Group resources: Data lineage, data quality and data contracts as security controls · AI Training Data Provenance and Copyright Exposure Guide · AI Model Provenance and Watermarking Guide · AI Supply Chain and AI-BOM Guide · Data Classification Guide