Apache Iceberg’s governance gap in multi-engine data access

At a glance

What this is: Trust3 argues that Apache Iceberg’s zero-copy storage model creates a governance gap when multiple engines apply different access, masking, and audit rules to the same data.

Why it matters: IAM, data security, and AI governance teams need consistent policy enforcement across engines and agents, or shared storage becomes easier to consume but harder to control.

Context

Apache Iceberg is a shared storage layer, not a governance layer. It lets multiple compute engines read the same data without copying it, but the article shows that access control, masking, and audit logic still live in the surrounding tools. That creates policy drift when Spark, Snowflake, Databricks, Flink, and AI workloads all touch the same sensitive records.

For identity and access teams, the risk is not the storage format itself. The risk is that entitlement decisions, masking outcomes, and approval paths fragment by engine, by workflow, and by workload type. In environments with human analysts and AI agents using the same dataset, the control problem shifts from where data sits to who or what can use it, under what context, and with what traceability.

Key questions

Q: What breaks when Apache Iceberg governance is left to each compute engine?

A: The same table can produce different access outcomes depending on which engine reads it. Masking, approvals, and audit trails fragment, so users may see raw data in one tool and protected data in another. That inconsistency weakens compliance, slows investigations, and creates hidden paths to sensitive records.

Q: Why do externally shared cloud files increase the risk of sensitive data exposure?

A: Externally shared files expand the number of people and systems that can reach sensitive content, which increases the chance of accidental disclosure or misuse. The risk is highest when organisations do not know exactly what data is in those files. Visibility and classification let teams identify exposed records, restrict access appropriately, and reduce the chance that sharing workflows become exfiltration paths.

Q: How should security teams govern AI agents that reason across multiple data platforms?

A: Security teams should govern the meaning layer, not just the access layer. That means defining shared business terms, lineage, and quality signals centrally, then making sure agents retrieve that context at runtime across every platform they touch. Without that control, the same agent can reach different conclusions from the same data.

Q: What is the difference between local engine controls and central policy enforcement?

A: Local engine controls apply rules inside one platform, while central policy enforcement decides access once and pushes the result consistently across all engines. The first approach is easier to deploy but fragile at scale. The second reduces drift, improves auditability, and gives governance teams one place to define intent.

Technical breakdown

Why multi-engine Iceberg deployments drift from a single policy model

Apache Iceberg separates storage from compute, which is useful for portability but awkward for security governance. Each engine consumes the same tables through its own permission model, its own caching layer, and often its own metadata interpretation. That means one platform may enforce row or column masking while another returns raw fields, even though both are reading the same underlying data. The architecture also encourages local control planes to accrete around each engine, so policy becomes an implementation detail rather than a shared governance decision.

Practical implication: teams need a central policy decision point if they want identical access outcomes across engines.

Why masking and audit controls fail when enforcement is engine-specific

Masking only works when the decision to hide data is applied consistently at read time, not after the fact. In a multi-engine stack, a user can see masked values in one tool and unmasked values in another if the policy logic is duplicated or translated imperfectly. Audit logging has the same problem. If logs are fragmented across lakehouse, catalogue, and query engines, investigators must reconstruct the access story manually, which slows response and weakens evidence quality. This is a governance consistency problem, not just a logging problem.

Practical implication: normalise masking and audit decisions across the full query path, not inside each platform in isolation.

How AI agents change the control problem for open data layers

AI agents and LLM applications turn shared data access into a runtime governance issue. These systems can generate requests dynamically, follow prompts that change by session, and pull data without a human operator making each decision. If the surrounding stack only recognises named users or static roles, the agent may inherit broad access that was never intended for machine use. The article’s point is that governance has to understand context such as purpose, sensitivity, and workload identity, otherwise agent access becomes a hidden path to raw data exposure.

Practical implication: treat AI agents as governed consumers with explicit context and purpose controls before they are allowed to query shared data.

Threat narrative

Attacker objective: The objective is to reach sensitive records through the weakest engine-specific control path and extract data that should have been masked or denied.

1. Entry occurs when multiple engines are allowed to query the same Iceberg data without a central enforcement layer, creating inconsistent control boundaries.
2. Credential or entitlement sprawl follows as separate engines and tools retain overlapping access, including dormant accounts and broad business-intelligence permissions.
3. Escalation happens when analysts or AI workloads reach raw fields through an engine that does not apply the same masking or context checks as the others.
4. Impact is exposure of sensitive data, slower incident response, and governance failure across a shared data estate.

Breaches seen in the wild

• Snowflake breach: Snowflake breach compromised Ticketmaster, Santander and others via cloud credential abuse.
• CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Policy drift is the real security failure in multi-engine data architectures. Apache Iceberg can standardise storage, but it does not standardise enforcement, so organisations end up with different access outcomes for the same dataset. That turns governance into a per-engine implementation problem, which is exactly where control inconsistency begins. Practitioners should treat policy drift as a first-class risk in shared data platforms.

Context-aware governance is now required for machine consumers as well as people. The article’s AI agent example shows why human-centric access models no longer cover the whole control surface. When agents can request data dynamically, static role logic and manual approvals miss the real decision point. Identity teams need to govern workload identity, purpose, and sensitivity together, not as separate reviews.

Central auditability matters more than local engine controls. Fragmented logs across lakehouse and query tools make it difficult to prove what was accessed, by whom, and under which policy. That is a governance and accountability problem, not just an operations issue. The practical conclusion is that shared data estates need a single evidentiary view if compliance and response are to remain credible.

ABAC and PBAC become the natural control model for open data layers. Role-only access is too blunt when one engine serves an analyst, another serves a fraud workflow, and a third serves an AI application. Attribute and purpose checks align better with how shared data is actually consumed, but only if the policy engine sits above the execution engines. Practitioners should design for policy portability, not just data portability.

What this signals

Policy drift is the hidden failure mode in open data platforms. When the same dataset is governed differently by separate engines, compliance becomes an outcome of architecture accidents rather than policy design. For practitioners, the priority is to place decision-making above the engines so masking, logging, and access denial cannot diverge as the stack grows.

AI-ready data estates need workload identity and purpose controls, not just human IAM. Once LLMs and autonomous agents can query shared storage, the control question changes from who logged in to what the workload is allowed to do with specific data. That pushes governance teams toward context-aware enforcement, which is more demanding but unavoidable in mixed human and machine environments.

For practitioners

• Define a central policy decision layer Place access, masking, and denial decisions above Spark, Snowflake, Databricks, Flink, and other engines so the same data request is evaluated once, not reinterpreted per platform.
• Unify masking rules across all query paths Test the same sensitive field through every engine that can reach it and verify that masked output, audit evidence, and denial behaviour are identical.
• Separate human and AI agent access paths Require explicit workload identity, purpose, and sensitivity context before LLMs or autonomous agents can query shared datasets.
• Rebuild audit trails around the shared dataset Correlate catalogue, engine, and governance logs into one access narrative so investigators can prove which policy applied to each read.
• Review dormant and broad BI entitlements Inventory long-unused analytics accounts and remove privileges that survive after role changes, especially where multiple engines share the same storage layer.

Key takeaways

• Apache Iceberg can simplify data sharing, but governance fragments quickly when each engine enforces its own access and masking logic.
• The exposure risk comes from inconsistent enforcement, fragmented audit trails, and dormant privileges across a shared data estate.
• Security teams need central policy decisioning and workload-aware controls if they want open storage without open exposure.

Key terms

• Policy Drift Detection: Policy drift detection is the process of identifying when an access policy no longer matches the intended rule or the current business context. It helps teams catch unauthorized changes, stale permissions, and exceptions that have quietly become the new normal, so governance stays aligned with actual identity and app usage.
• Central Policy Enforcement: Central policy enforcement is a governance pattern where access decisions are made once and applied consistently across multiple consumers. It reduces fragmentation by separating intent from execution, which is especially important when many engines or AI workloads can read the same data.
• Context-Aware Access Review: A decision process that evaluates access requests using request intent, existing entitlements, resource sensitivity, and operational evidence. In identity programmes, it reduces blind approvals by forcing reviewers or automation to weigh the real circumstances behind the request, not just the ticket itself.
• Unified audit trail: A single evidence stream that links identity proofing, authentication, consent, and executed actions across human and non-human actors. It matters because split logs prevent teams from reconstructing what the agent was allowed to do and what it actually did.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to connect access governance, machine identity, and operational control across modern environments.