OpenLineage

OpenLineage is an open standard for capturing and sharing data lineage events across tools and platforms. It lets processing systems emit consistent metadata about jobs, datasets, and transformations, which governance tools can ingest to build a more complete view of data movement and change.

Expanded Definition

OpenLineage is a vendor-neutral standard for emitting machine-readable lineage events, such as job runs, dataset inputs and outputs, and transformation metadata. In NHI security and governance, that matters because the systems that move data are often operated by non-human identities, including CI/CD agents, orchestration services, and transformation pipelines.

Its value is not just traceability, but interoperability. Instead of every platform inventing its own lineage format, OpenLineage creates a shared event model that downstream observability, catalog, and governance tools can consume. That makes it easier to answer who changed what, where a dataset came from, and which automated identity performed the action. The open standard is closely aligned with broader governance patterns in the NIST Cybersecurity Framework 2.0, especially visibility and risk management. Usage in the industry is still evolving, and implementation depth varies across vendors, so lineage completeness should be treated as an operational capability rather than a guaranteed outcome.

The most common misapplication is treating OpenLineage as proof of data trustworthiness, which occurs when organisations confuse event emission with validated identity, access, and transformation controls.

Examples and Use Cases

Implementing OpenLineage rigorously often introduces instrumentation and integration overhead, requiring organisations to weigh richer governance visibility against added engineering effort across pipelines and platforms.

• A scheduled ETL workflow emits lineage events for each transformation step, allowing governance teams to trace dataset drift back to the exact job and run.
• A data platform uses lineage metadata to show which service account published a sensitive table, linking operational activity to the identity behind it.
• A CI/CD pipeline for analytics code emits OpenLineage events so downstream controls can detect when a deployment changed schema, ownership, or processing logic.
• A security review correlates lineage with secret exposure findings, helping investigators see whether a compromised API key touched upstream or downstream data assets.
• An organisation maps lineage into its broader NHI inventory to understand which automation identities have the most impactful data movement privileges, a concern highlighted in the Ultimate Guide to NHIs.

OpenLineage is especially useful when paired with platform-native telemetry and identity controls such as NIST Cybersecurity Framework 2.0 outcomes for asset visibility and access monitoring.

Why It Matters in NHI Security

Lineage becomes an identity-security issue because automated data movement is rarely anonymous. If a pipeline, scheduler, or transformation engine is compromised, OpenLineage can help investigators reconstruct what was accessed, changed, or propagated. That makes it useful for incident response, segregation of duties, and governance evidence, especially where NHIs are implicated in broad data operations.

This matters because NHI sprawl is already severe: Ultimate Guide to NHIs reports that NHIs outnumber human identities by 25x to 50x in modern enterprises, and 97% carry excessive privileges. In that environment, lineage metadata can help show not only where data went, but which machine actor moved it and whether the action fit expected authority. It also supports governance alignment with the NIST Cybersecurity Framework 2.0 by improving traceability and response readiness.

Organisations typically encounter the need for lineage evidence only after a suspicious pipeline run, at which point OpenLineage becomes operationally unavoidable to address.