Apache Iceberg’s governance gap in multi-engine data access

At a glance

What this is: This is an analysis of how Apache Iceberg’s zero-copy data sharing can create governance drift when multiple engines enforce access, masking, and audit rules separately.

Why it matters: It matters to IAM, data security, and AI governance teams because shared data layers can expose the same identity and policy gap across human users, service accounts, and AI agents.

👉 Read Trust3's analysis of Apache Iceberg governance in multi-engine data access

Context

Apache Iceberg improves data portability by letting multiple engines read the same storage layer, but that convenience also breaks the assumption that one control plane can govern access everywhere. When masking, auditing, and entitlement decisions are split across lakehouse tools, query engines, and AI applications, policy drift becomes the default.

The identity angle is real because this is not only a data architecture issue. Human users, dormant accounts, service accounts, and AI agents can all inherit inconsistent access paths unless governance is enforced centrally across the full access lifecycle.

Key questions

Q: How should security teams govern access across multiple data engines?

A: Use one policy authority for intent, then enforce it through each engine’s native controls. That gives you a consistent decision model without pretending Databricks, Snowflake, and other platforms behave identically. The key is to reconcile drift continuously so the enterprise has one answer to who can access what, regardless of the query path.

Q: Why does Apache Iceberg-style sharing create governance risk for identity teams?

A: Because the same dataset may be reachable by users, service accounts, and AI agents through different control paths. If identity and access decisions are not centralised, a person can be restricted in one engine but effectively privileged in another. That turns identity governance into a cross-platform consistency problem, not just an entitlement review.

Q: What breaks when masking rules are enforced separately in each data tool?

A: Masked and unmasked views can diverge, which means one engine may return raw data while another hides it. That breaks privacy expectations, complicates investigations, and makes policy evidence unreliable. Teams then have to prove control effectiveness by comparing logs and outputs across systems instead of trusting a single governance model.

Q: How should organisations control AI agents that query sensitive business data?

A: Treat agents as governed identities with bounded purpose, not as open-ended query clients. Apply attribute and purpose checks, restrict raw field access, and default to aggregation or masking unless the request is explicitly justified. That keeps AI usage within policy and prevents agents from becoming a shortcut around human approval controls.

Technical breakdown

Why zero-copy data sharing weakens security boundaries

Zero-copy storage keeps the data in one place while letting different engines access it through their own runtime logic. That improves interoperability, but it also means security decisions are no longer made in one control plane. If Spark, Snowflake, Databricks, and other tools each interpret policies differently, the organisation gets multiple versions of the truth for masking, auditing, and entitlement checks. In practice, the weakest policy path becomes the effective policy path, especially when users move between engines or when AI applications query the same dataset through different interfaces.

Practical implication: treat shared storage as a governance coordination problem, not a storage problem, and enforce one policy source across all engines.

How fragmented masking and audit logs create hidden exposure

Masking is only reliable when it is applied consistently at the point of access. In a multi-engine environment, one platform may show masked fields while another returns raw values, which defeats the purpose of data classification. Audit gets harder for the same reason. If logs are spread across engines, incident response must reconstruct who accessed what from separate systems, which slows investigation and obscures accountability. The architecture also becomes brittle when former employee accounts or broad business intelligence permissions remain active across only part of the stack.

Practical implication: unify masking policy and audit collection across query engines before expanding data sharing to additional teams or workloads.

What context-aware control adds for AI and agentic access

AI applications do not just consume data, they can transform prompts into runtime requests against sensitive datasets. That makes context-aware authorisation more important than static role checks alone. Purpose-based access control, sometimes paired with attribute-based logic, can evaluate role, purpose, location, and sensitivity before data is released. For AI agents, this matters because the agent may request broad data to satisfy a task unless the control layer restricts fields, masks identifiers, or reduces the result to aggregates. The governance challenge is to make machine-driven access behave like a bounded business process rather than an open-ended query channel.

Practical implication: apply purpose and attribute checks to AI-driven data requests so agents cannot bypass human-oriented masking rules.

Threat narrative

Attacker objective: The objective is to obtain sensitive data through the least governed access path and keep that access invisible to central oversight.

1. Entry occurs when broad data access is granted through multiple engines that do not share a single governance model. Credentialed users, service accounts, or AI agents can reach the same dataset through different paths.
2. Escalation happens when each engine applies its own masking, entitlement, or logging rules, allowing raw fields and persistent permissions to survive in one part of the stack even when another part restricts them.
3. Impact follows when sensitive data, including PII, is exposed to users, analytics workflows, or AI applications without consistent control or reliable audit evidence.

NHI Mgmt Group analysis

Policy fragmentation is the real risk in open data architectures. Apache Iceberg-style sharing does not fail because storage is open, it fails because security decisions are duplicated across engines that do not share state. That creates inconsistent masking, uneven entitlement enforcement, and audit gaps that no single team can see end to end. The governance failure is architectural, not procedural, and the practitioner conclusion is to collapse policy drift before expanding the data plane.

Open data layers now expose a named problem: control-plane drift. This is the condition where the storage layer is unified but the identity, masking, and audit controls are not. In that state, a user can be low-risk in one engine and effectively privileged in another, which breaks trust in the entire analytics estate. The relevant practitioner response is to treat policy compilation and enforcement as a shared runtime service, not a per-tool configuration task.

AI access turns data governance into identity governance. When autonomous agents query customer or transaction data, the organisation is no longer just authorising a person or a service account. It is authorising a software decision process that can expand, summarise, and repurpose data at runtime. That makes this a genuine intersection with NHI governance, because agent identities and workload identities need the same lifecycle, purpose, and scope discipline as privileged human access. The practitioner conclusion is to govern agent access as a first-class identity problem.

Dormant access and over-broad BI entitlements are governance failures, not cleanup issues. The article’s strongest warning is that access sprawl accumulates quietly while teams focus on throughput and self-service. Once the data layer becomes shared across multiple engines, legacy accounts and broad analytical permissions persist unless lifecycle controls are enforced centrally. That maps closely to the broader NHI problem space, where standing access and weak offboarding turn convenience into latent exposure. The practitioner conclusion is to bind access review, revocation, and audit to the control plane, not to individual tools.

What this signals

Control-plane drift: The fastest way to lose governance in shared data architectures is to let each engine interpret policy independently. For identity and data teams, the next programme maturity step is to unify authorisation, masking, and logging around a single runtime decision point, not around individual tools.

AI usage will force data governance teams to think like IAM teams. Once agents can query sensitive datasets, access scope, purpose, and revocation need to be defined with the same discipline used for privileged identities and workload credentials.

The pressure point will be lifecycle control, not just policy design. In environments where dormant accounts, broad analytical access, and machine-driven access all coexist, organisations need continuous review of who or what can reach raw fields, and why.

For practitioners

• Centralise policy enforcement across engines Define access, masking, and audit rules once and compile them into every engine that can query the shared data layer. Do not rely on separate engine-level policies to stay aligned over time.
• Unify audit trails before expanding access Join logs from query engines, lakehouse controls, and AI applications into one investigation path so incident response can reconstruct access without manual correlation across systems.
• Apply purpose-based checks to AI requests Require runtime evaluation of role, purpose, location, and sensitivity before agents or users can retrieve raw fields, especially for customer, employee, or financial data.
• Review dormant and over-broad accounts Identify inactive user accounts, service accounts, and BI entitlements that still span multiple engines, then revoke or narrow them at the central governance layer.

Key takeaways

• Open data sharing does not remove governance requirements. It moves the control problem into the space between engines, where inconsistency is harder to detect.
• The biggest risk in this pattern is not storage exposure alone. It is fragmented masking, fragmented logs, and fragmented identity decisions across humans, service accounts, and AI agents.
• Practitioners should centralise policy enforcement, audit, and lifecycle revocation before scaling shared data access to more teams or more autonomous workloads.

Key terms

• Hybrid Identity Control Plane Drift: Hybrid identity control plane drift is the gap that appears when different systems enforce access, review, and revocation through separate administrative models. It leads to inconsistent decisions about privilege and session handling, which weakens governance even when individual tools are functioning correctly.
• Intent-Based Access Control: An access control model that evaluates not just what an agent is requesting, but the inferred intent and context behind the request — granting or denying access based on whether the action aligns with the agent's declared purpose.
• Shared Context Layer: A shared context layer is the common record or state object that multiple agents use to coordinate work. It keeps reasoning, handoffs, and decisions visible so operators can audit actions, intervene safely, and avoid hidden state scattered across private memory or separate channels.
• Dormant access: Dormant access is an entitlement that remains technically valid even though the subject no longer uses it for its intended purpose. In identity governance, dormant access is dangerous because it preserves privilege, complicates review, and often survives well past the business need that created it.

What's in the full article

Trust3's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step governance hub workflow for policy-driven access requests across Iceberg-connected engines
• Runtime policy compilation approach for Spark, Snowflake, Databricks, and Flink access enforcement
• Unified context layer logic for masking PII and handling AI prompt requests
• Policy examples for attribute and purpose-based access control in regulated data use cases

👉 Trust3's full post covers the control-plane design, policy enforcement flow, and AI access handling details.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management through a practitioner-led framework. It helps identity and security teams connect access control to the broader security programme they operate.