Databricks governance needs a policy layer above native catalogs

At a glance

What this is: Trust3 argues that Databricks governance works inside the platform, but enterprise data estates need a policy layer above native catalogs to keep access decisions consistent across multiple engines.

Why it matters: IAM, data governance, and cloud security teams need a reconciled control plane because heterogeneous platforms create policy drift, fragmented audits, and inconsistent access decisions.

Context

Governance fails when policy intent and enforcement are split across multiple data engines that each maintain their own access model. In a mixed estate, the security problem is not whether a single catalog works inside one platform, but whether access decisions remain consistent when the same data is read through different control planes.

This article sits at the intersection of data governance and identity governance. The important issue for IAM and GRC teams is not just who can read a table, but how policy, entitlement, and audit evidence stay aligned when Databricks, Snowflake, and other engines all participate in access enforcement.

Key questions

Q: What breaks when native catalogs are the only governance authority across engines?

A: Policy drift becomes inevitable because each engine applies its own access model and audit trail. The practical failure is not a single bad grant, but inconsistent enforcement across platforms that claim to govern the same data. Once that happens, teams lose a defensible answer to who accessed what and why.

Q: Why does heterogeneous data infrastructure increase governance risk?

A: Because the same dataset may be read through multiple engines that do not share the same policy semantics. That creates reconciliation gaps, inconsistent entitlements, and audit evidence that no longer matches actual enforcement. The more engines involved, the more important a single policy intent layer becomes.

Q: How do teams know whether reconciled governance is actually working?

A: Look for consistent policy translation, documented exceptions where an engine lacks a native equivalent, and drift reports that stay close to zero over time. If audits keep finding different access outcomes for the same asset, the governance model is descriptive, not enforced.

Q: What should security and data teams do when control has to span Databricks and other engines?

A: Keep native enforcement in each platform, but centralise policy intent and reconciliation above them. That lets teams govern access consistently without pretending one catalog can own the whole estate. The goal is not uniform tooling, but uniform decision logic with local enforcement.

Technical breakdown

Policy administration point versus policy enforcement point

A policy administration point defines intent, while a policy enforcement point applies it inside each runtime. In a multi-engine data estate, that split matters because the same table may be queried through different engines, each with its own native authorization model. The control plane has to compile one policy into multiple engine-specific constructs without becoming the decision engine at query time. That keeps enforcement local, preserves performance, and avoids central coupling. The hard part is translation fidelity, because row filters, masks, tags, and grants do not map perfectly across systems.

Practical implication: Treat the policy layer as a compiler and reconciliation engine, not as a runtime bottleneck.

Why native catalogs stop at platform boundaries

Native catalogs are strong inside a single platform because they unify metadata, lineage, and fine-grained access controls in one place. They fail as an enterprise anchor when data is distributed across engines that do not share the same policy object model. In that situation, each platform becomes a separate governance island, and the enterprise loses a single answer to who-can-do-what. The result is not just administrative duplication. It is inconsistent access semantics, uneven evidence, and policy drift between engines that all claim to govern the same asset.

Practical implication: Use native catalogs for local enforcement, but do not let them become the only governance authority across the estate.

Reconciliation is the control that keeps policy honest

A reconciled control plane checks whether compiled policy in each engine still matches the authoritative intent. That means detecting drift, surfacing unsupported controls, and showing where an engine can only approximate the intended rule. This is especially important for regulated data because an access model that exists only in documentation is not audit evidence. Reconciliation is what turns distributed enforcement into something defensible: not perfect uniformity, but verified consistency with explicit exceptions. Without it, teams confuse policy distribution with policy assurance.

Practical implication: Build drift detection and exception reporting into governance so audits reflect actual enforcement, not policy aspirations.

NHI Mgmt Group analysis

Policy drift is the real governance failure in heterogeneous data estates: the problem is not that native catalogs exist, but that they multiply into separate sources of truth as soon as data spans engines. That makes the enterprise answer to access questions dependent on which platform answered them last. For IAM and data governance teams, the practical issue is reconciled control, not catalog feature depth.

Databricks-centric governance cannot be the enterprise control point: Unity Catalog is a valid enforcement authority inside Databricks, but it becomes a dependency risk when the wider estate includes Snowflake, Lake Formation, BigQuery, and Iceberg. The governance model has to survive platform diversity, so policy intent must sit above any single engine. That is the only way to keep ownership from collapsing into vendor locality.

Reconciled control planes are becoming the missing layer in modern access governance: enterprises need one place to define intent and many places to enforce it natively. That pattern aligns with cloud governance, IAM, and audit discipline because it separates decision-making from execution while preserving local control. The named concept here is reconciled control-plane governance: a policy model that compiles once, enforces locally, and continuously checks for drift.

Standing access is too weak a model for multi-engine governance: the article’s own examples point to purpose-based and expiry-based access as the better mental model. That shifts governance closer to ABAC-style intent, where policy reflects context rather than static role assignment alone. For practitioners, the conclusion is clear: if access cannot be explained consistently across engines, it is not governed, only distributed.

AI workloads raise the stakes for this pattern: the article’s final note is the right warning, because agents and automated workloads will traverse these same estates faster than human reviewers can reconcile them. That makes policy translation and drift control foundational, not optional. The implication for security and data teams is to design governance for machine speed before AI traffic multiplies the inconsistency.

What this signals

Reconciled control-plane governance: the enterprise problem is no longer catalog design inside a single platform, but policy consistency across engines that each enforce differently. Security and data leaders should treat drift detection and exception handling as first-class governance controls, because cross-platform estates make inconsistency the default.

Multi-engine estates force a harder IAM question: where does authority live when enforcement is distributed? The answer cannot be one vendor platform, because governance now has to survive platform diversity, workload mobility, and machine-speed access patterns. Teams should design for policy translation and evidence reconciliation before AI workloads add more dynamic access to the estate.

For practitioners

• Define policy once and compile it per engine Separate the policy administration point from the policy enforcement point so Databricks, Snowflake, Lake Formation, and other engines enforce native controls while one authority defines intent.
• Add reconciliation checks for policy drift Compare authoritative policy intent with what each engine actually enforces and flag unsupported rules, stale translations, and mismatched entitlements.
• Model access with purpose and expiry Replace standing grants where possible with purpose-scoped and time-bounded access so entitlement reviews can verify why access exists, not just who has it.
• Plan for engine outage behavior Validate that a control plane outage blocks new policy changes without opening access, and document the fail-static behavior for audit and incident response.

Key takeaways

• Native catalogs are useful inside a platform, but they do not solve enterprise governance when the same data is accessed through multiple engines.
• The operational risk is policy drift, inconsistent entitlements, and audits that no longer match actual enforcement across the estate.
• A reconciled control plane gives teams one place to define intent and many places to enforce it locally, which is the only durable pattern for heterogeneous environments.

Key terms

• Policy Administration Point: A policy administration point is the control layer where authorization rules are created, reviewed, tested, and distributed. In practice, it acts like an identity policy plane, so its change management, ownership, and auditability matter as much as the policy language itself.
• Policy Enforcement Point: A policy enforcement point is the control that applies an authorization decision at the place where an action occurs. In distributed systems, it may sit inside an API gateway, application, or workflow engine, and it depends on a consistent decision format to avoid bespoke integrations.
• Reconciled Control Plane: A governance layer that keeps one authoritative policy intent and checks that each engine enforces it correctly. It does not replace native controls. Instead, it coordinates them, detects drift, and exposes exceptions where a platform cannot implement the intended rule exactly.
• Policy Drift Detection: Policy drift detection is the process of identifying when an access policy no longer matches the intended rule or the current business context. It helps teams catch unauthorized changes, stale permissions, and exceptions that have quietly become the new normal, so governance stays aligned with actual identity and app usage.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and identity lifecycle thinking that helps teams manage governed access at scale. It gives IAM, security, and data leaders a common language for policy, ownership, and control across modern estates.