Data lineage, data quality and data contracts as security controls: map, test and police how data moves

Most security teams can say who may read a table, but very few can say where its sensitive columns go next, who changed them on the way, or whether the copy an AI pipeline reads is still the data that was approved. Data lineage, data quality checks and data contracts were built by data engineering teams for reliability, yet the same metadata answers the questions an investigator, a privacy officer and an access reviewer ask. This guide shows how to use them as controls: lineage as evidence of data flow, column-level masking as a property you can verify, quality assertions as integrity checks, contracts as the place where classification, roles and retention are agreed, and lineage events as a feed for spotting risky data movement. It is written for data platform owners, security architects and detection engineers who already have a catalogue or warehouse and want it to carry security weight. Software build provenance is out of scope and is covered in the SLSA and Build Provenance Guide.

Key takeaways

• Lineage is data-flow evidence that is generated by the pipelines themselves, so it stays current in a way a hand-drawn diagram never does, but every platform has gaps (renames, UDFs, path references, retention windows) that you must know before relying on it.
• Column-level lineage is the security grade because classification, masking and minimisation decisions live at column level; table-level lineage tells you a table fed a dashboard, not whether the national insurance number survived the trip.
• Data quality checks double as integrity controls: a sudden change in row count, null rate, distinct values or freshness is often the first visible sign of a broken job, a tampered load or a bulk deletion.
• Data contracts are where security requirements become machine-readable: classification, critical data elements, approved roles, approvers, retention and quality thresholds sit next to the schema and can be enforced in CI before a change ships.
• Lineage and access history together reveal risky movement: new edges from restricted datasets to external stages, exports, unmanaged workspaces or AI stores are the signal; the identities behind those edges are mostly service accounts and need owners.

Why lineage, quality and contracts belong in the security programme

Security controls for data usually focus on the store: encryption, grants, row and column policies. The weak point is movement. Sensitive data is copied into staging schemas, feature tables, extracts, BI datasets and vector stores, and each copy can lose the classification and masking of the original. The Data Governance and Data Catalogue Security Guide covers classification, catalogue-driven access and securing the catalogue; this page covers the three disciplines that follow data after it leaves the governed table.

• Data lineage: the recorded relationship between inputs, the job or query that transformed them and the outputs, at table and ideally column level. It answers "where did this come from" and "what depends on this".
• Data quality: measured properties of a dataset against expectations. The Open Data Contract Standard (ODCS) lists the dimensions accuracy, completeness, conformity, consistency, coverage, timeliness and uniqueness. For security, unexpected deviations are integrity signals.
• Data contracts and data products: a versioned agreement between a producer and consumers covering schema, quality rules, roles, service levels and ownership. ODCS (current version v3.2.0, published by Bitol under the LF AI & Data Foundation) is the open format most tools now read.
• Risky data movement monitoring: using lineage and access history to detect flows that should not exist, such as restricted columns appearing in an unmasked output or bulk copies to an external location.

Regulation pulls in the same direction. Under UK GDPR Article 5(1)(d) the ICO expects every reasonable step to keep personal data accurate, with more checking where data drives higher-impact decisions, and recommends recording where information came from when you cannot verify it yourself. That is lineage and quality by another name. Article 30 records of processing and data protection impact assessments both depend on knowing where personal data flows; see the Data Mapping and ROPA Guide.

How lineage is captured: OpenLineage and platform-native lineage

There are two capture models. Platform-native lineage is recorded by the warehouse or catalogue as queries run. OpenLineage is an open specification (JSON Schema plus an OpenAPI definition for HTTP transports) that schedulers and engines use to emit run events to any compatible backend.

The OpenLineage model in brief

• Run event: an observed state of a job run. The specification requires at least a START event and a COMPLETE, FAIL or ABORT event; metadata is additive across events.
• Job, Run and Dataset: a job has a unique name in a namespace (assigned by the scheduler), a run is identified by a UUID, and a dataset is named from its physical location (for example host, database, schema and table) within a datasource namespace.
• Facets: named pieces of metadata attached to runs, jobs or datasets. Emitting a facet with the same name replaces the previous one. Custom facets must carry a project prefix so they cannot collide with standard ones.
• Security-relevant standard facets: sourceCodeLocation (repository and git SHA of the job), sql (the query text), ownership (job and dataset owners), schema, dataSource, lifecycleStateChange (create, drop, overwrite, rename, truncate), columnLineage, dataQualityMetrics, dataQualityAssertions, inputStatistics and outputStatistics (row count and bytes).
• Column lineage facet: maps each output field to the input fields used to compute it, with a transformation type of DIRECT or INDIRECT, a subtype and a boolean masking flag that states whether the transformation masks the data. Dataset-level dependencies (filters, joins, grouping, window functions) are recorded separately, which matters because a filter on a sensitive column is itself a use of that column.

Platform-native lineage: what each records and where it falls short

Two practical rules follow. First, export lineage you need as evidence into your own store, because 30 days or one year is shorter than most investigation and audit look-backs. Second, treat any manually created lineage edge as a change to a security record: it should need curation rights, a ticket and an audit log entry, because a falsified lineage graph can hide a flow.

Using lineage as a security control

• Classification propagation: walk column lineage downstream from every column tagged restricted or personal, and flag any output column with a DIRECT, unmasked dependency that lacks the same tag or a masking policy. This is the single highest-value lineage check. See the Data Classification Guide.
• Masking verification: where a transformation claims to mask (hashing, tokenisation, truncation), check the column lineage masking flag or the platform's policy reference (for example Snowflake policies_referenced) rather than trusting the job name. See the Tokenisation and Data Masking Guide.
• Blast radius in incidents: when a source is found poisoned, corrupted or exposed, query downstream lineage to list every derived table, dashboard, model and extract.
• Access review scope: a person or workload with read access to a derived table effectively has read access to the restricted source columns it inherited. Include derived datasets in reviews of sensitive sources.
• Erasure and retention: deletion requests and retention rules must reach copies. Downstream lineage is the list of places to purge; see the Data Retention and Secure Deletion Guide.
• AI training and retrieval sources: lineage from source tables to feature stores, training sets and vector stores shows which models were built on which data, which is what you need when a source is withdrawn or found tampered. See the AI Data Poisoning and Backdoors Guide.

Data quality checks as integrity controls

Quality frameworks express expectations as rules with thresholds. ODCS defines rule types text, library, SQL and custom (for engines such as dbt tests, Great Expectations or Soda), with library metrics nullValues, missingValues, invalidValues, duplicateValues and rowCount and operators such as mustBe, mustBeLessThan and mustBeBetween. AWS Glue Data Quality uses DQDL, with rule types including IsComplete, IsUnique, ColumnValues, RowCount, RowCountMatch, ReferentialIntegrity, SchemaMatch, DataFreshness, CustomSql and DetectAnomalies, and dynamic rules such as RowCount > avg(last(5)) that compare a run with history. The security use is to choose rules that would fire on tampering, not just on bugs.

Route quality failures on restricted or critical data elements to the security queue as well as the data team. A rule failure on a payments ledger is a possible integrity incident until shown otherwise. The Data Integrity Protection Against Destructive Attacks Guide covers backups and immutability for the cases where a check fires too late.

Data contracts and data products: putting security in the agreement

A data contract makes the producer accountable for what consumers receive. ODCS sections include fundamentals, schema, data quality, support channels, team, roles, service levels, servers, authoritative definitions and tags. Several fields carry security meaning directly.

Decision guidance: small organisations with one warehouse can get most of the value from platform-native lineage, classification tags and a handful of quality rules on critical tables; contracts can be a YAML file per important dataset reviewed in pull requests. Large organisations with many domains, a data mesh or several platforms need contracts as the interface between teams and OpenLineage (or a catalogue that imports it) to join lineage across engines. Use contracts where producers and consumers are different teams, not for single-team internal tables.

Monitoring risky data movement

Lineage tells you a flow exists; access history tells you who drove it. Combined, they let you alert on movement rather than only on access. The Snowflake customer attacks of 2024 are the reference case: the attackers listed tables, created temporary stages and copied whole tables out with ordinary SQL, so the activity looked like normal use. See the Snowflake Breach 2024 and the Snowflake Security Guide.

Platform latency matters: Snowflake access history can lag up to 180 minutes and Google lineage up to 24 hours, so these are detective controls. Pair them with preventive egress controls; see the Data Loss Prevention (DLP) Guide and the Data Exfiltration Guide.

Where identity fits

Almost every edge in a lineage graph is created by a non-human identity: an orchestrator's service account, a transformation tool's warehouse user, a BI connector or an ML pipeline's workload identity. That makes lineage a usable inventory of what each machine identity actually does with data. Use it to right-size grants (a job that only ever reads three tables does not need schema-wide read), to give every job namespace a human owner (the OpenLineage ownership facet and the ODCS team section are the natural place), and to alert when an identity's lineage footprint changes. Strong authentication matters too: the Snowflake attacks used password-only accounts. See the Service Accounts in Snowflake Guide and the Database Credential and Access Security Guide. Lineage emitters send SQL text and schema to a backend, so protect the lineage store like the catalogue.

Implementation roadmap

1. Pick the scope: list the 10 to 30 datasets holding restricted, personal or critical data and their known consumers. Start there, not with the whole estate.
2. Turn on native lineage and access history: enable the Data Lineage API, Unity Catalog system tables or Snowflake access history as applicable, and confirm which jobs actually produce lineage. Record known gaps (UDFs, path references, renames, load jobs) per platform.
3. Add OpenLineage where native capture stops: configure the scheduler and engine integrations to emit to your backend, include sourceCodeLocation, ownership and columnLineage, and prefix any custom facets.
4. Export and retain: copy lineage and access history to your own store with retention that meets audit and investigation needs, and restrict who can create manual edges.
5. Write contracts for the in-scope datasets: schema with classification and critical data elements, roles and approvers, retention and quality rules. Validate them against the ODCS JSON Schema in CI.
6. Run the checks in audit mode: run classification propagation, masking verification and movement rules for two to four weeks, alerting only to the data platform team, and tune out legitimate flows.
7. Enforce: fail builds on contract breaches, route critical quality failures and movement alerts to the SOC, and block deployment of transformations that drop masking.
8. Roll back safely: keep enforcement switchable per dataset; if a rule blocks a critical pipeline, revert to audit for that dataset, open a ticket and fix the rule within an agreed time rather than deleting it.

Detection and measurement

• Lineage coverage: percentage of in-scope restricted datasets with column-level lineage to every known consumer. Target close to full coverage for the in-scope list.
• Classification propagation gaps: count of downstream columns with an unmasked DIRECT dependency on a restricted column but no matching tag or policy. Trend should fall to zero.
• Contract coverage: share of in-scope datasets with a validated contract including classification, roles and retention.
• Critical quality rule failures: number per month and median time to triage, split by whether the cause was a bug or a security event.
• Unexplained movement alerts: movement alerts closed as unapproved flows, and time from flow creation to detection.
• Owned job identities: percentage of lineage-producing service accounts and job namespaces with a named human owner.

Common failure patterns

• Trusting lineage as complete: UDFs, path-based reads, renamed objects and unsupported jobs leave holes, so a flow that does not appear in the graph is not proof that it does not exist.
• Short retention used as evidence: with 30-day lineage retention, the graph for an incident discovered in month three is gone. Export it.
• Table-level lineage only: the graph shows the customer table feeds the marketing extract but not that the full email column went with it, so reviewers approve a flow they would have refused.
• Quality alerts owned only by data engineering: a row count drop on a ledger is closed as a pipeline issue and nobody asks whether it was a deletion. The Snowflake attacks showed how bulk activity can pass as ordinary use when nobody watches volume.
• Contracts without classification: schema and SLAs are agreed but the sensitive fields are not marked, so the contract enforces everything except security.
• Unowned pipeline identities: long-lived service accounts with broad read create lineage that nobody reviews.

Framework mapping

Practitioner checklist

• List the restricted and critical datasets in scope and their known consumers.
• Enable platform-native lineage and access history, and document each platform's gaps and retention.
• Emit OpenLineage from schedulers and engines not covered natively, with column lineage, ownership and source code location.
• Export lineage and access history to a store with retention matched to audit and investigation needs.
• Run a classification propagation check on column lineage and fix every unmasked restricted dependency.
• Write ODCS contracts for in-scope datasets with classification, critical data elements, roles, approvers, retention and quality rules, validated in CI.
• Add tamper-sensitive quality rules (row count against history, referential integrity, freshness, pattern checks on masked columns) and route failures on critical data to security.
• Alert on new edges from restricted data to external stages, shares, unmanaged workspaces and AI stores.
• Give every lineage-producing service account and job namespace a named owner and review its footprint quarterly.
• Restrict and log manual lineage edits as changes to a security record.

Standards and references

• OpenLineage: OpenLineage Specification
• Bitol (LF AI & Data): Open Data Contract Standard (ODCS)
• ICO: Principle (d), Accuracy
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations
• Google Cloud: About data lineage (Knowledge Catalog)
• Databricks: View data lineage using Unity Catalog
• Snowflake: ACCESS_HISTORY view
• AWS: Data Quality Definition Language (DQDL) reference

Related NHI Mgmt Group resources: Data Governance and Data Catalogue Security Guide · Data flow mapping for security and privacy · ETL and ELT Pipeline Security Guide · SLSA and Build Provenance Guide · DSPM Implementation Playbook