Data Lake Security Guide: Access Models, Encryption and Governance

A data lake has no single front door: the same file can be reached through object storage, a catalog, several query engines and every pipeline that writes to it, and access is only as strong as the weakest of those paths. Modern lakes add open table formats such as Apache Iceberg and Delta Lake, REST catalogs that hand out storage credentials, and self-hosted object stores, each with its own security behaviour. This guide explains the four access models used in data lakes and lakehouses, how to choose between them, how encryption works at storage, file and column level, and how table format features such as snapshots and time travel affect governance and erasure. It is for data platform engineers, security architects and data governance leads. Cloud-provider specifics (Lake Formation, ADLS, BigLake) are covered in the Enterprise Data Lake Security Guide; this page focuses on the architecture decisions underneath them.

Key takeaways

• Pick one access model per zone and enforce it. Storage-direct, catalog-vended credentials, engine-enforced and gateway access give very different guarantees; mixing them silently on the same data is the most common design flaw.
• The catalog is now a credential issuer. Iceberg REST catalogs and Unity Catalog can vend short-lived, path-scoped storage credentials, which only works as a control if clients have no standing storage access of their own.
• Check that authorisation is actually on. Open-source Unity Catalog allows every request when server.authorization is not enabled, and AWS Lake Formation does not enforce permissions on resources that still grant IAMAllowedPrincipals.
• Deleted is not gone in a table format. Snapshots and time travel keep removed rows in older data files until snapshots are expired and files vacuumed, which matters for GDPR erasure and incident clean-up.
• Use the right encryption layer for the threat. Storage-side encryption protects media; Parquet modular encryption with column keys protects sensitive columns from anyone who can read the files but lacks the key.

Data lake architecture and the layers that hold access

Whether it runs on Amazon S3, Azure Data Lake Storage, Google Cloud Storage, HDFS or a self-hosted S3-compatible store, a data lake has five layers. Each can grant access, and each needs its own control.

The key observation is that file storage sits beneath every other layer. A column mask in the engine means nothing to a principal who can read the underlying Parquet files from the bucket. Every access model below is a different answer to the question "who is allowed to hold a storage credential?"

Lakes are also usually organised into zones (raw or landing, cleansed, curated, and sandbox, sometimes called bronze, silver and gold). The raw zone is often the most sensitive, because it holds data before minimisation, masking or tokenisation, yet it is frequently the least governed. Put zones in separate buckets, containers or accounts so that storage policy, encryption keys and logging can differ per zone.

The four access models

1. Storage-direct access

Users, jobs and tools hold IAM roles, service principal secrets, access keys or Kerberos identities that read storage directly, and table-level security is enforced only by convention. This is how most first-generation lakes were built. It is simple and fast, but authorisation is at file or prefix level only, and every consumer is effectively a storage administrator for the paths it can reach. Acceptable for a single team's sandbox; not acceptable for regulated or shared data.

2. Catalog-governed with credential vending

Clients authenticate to the catalog, the catalog checks table privileges, and then returns short-lived storage credentials scoped to that table's location. The Apache Iceberg REST catalog specification defines this through the X-Iceberg-Access-Delegation header, with two mechanisms: vended-credentials, where the server returns storage-credentials for S3, ADLS or GCS, and remote-signing, where the client asks the catalog to sign each storage request so it never holds a storage credential at all. Open-source Unity Catalog works the same way: a storage credential holds the server's cloud identity, an external location pairs a path with that credential, and the server issues temporary credentials scoped to a single table, volume or path. Its documentation is explicit that the catalog only becomes the gatekeeper if clients are denied direct storage access.

3. Engine-enforced fine-grained access

A trusted engine or service holds the storage credential and enforces row filters, column masks and tag policies on behalf of users, as Apache Ranger does for Hive and as cloud lake services do for their integrated engines. This is the only model that supports row and column controls. Its weakness is that the policy only holds where the engine is the sole reader; untrusted engines must receive either no access or full-table access only.

4. Gateway or share-based access

Consumers never see storage: they query through an API, a SQL endpoint or a data sharing protocol, and the provider controls what leaves. This is the right model for external parties and for AI agents that need answers, not files.

Choosing between them

• If the data is personal, financial or health data: engine-enforced or gateway access, with storage readable only by the engine's service identity.
• If several engines (Spark, Trino, Flink) must read the same tables: a catalog with credential vending, with standing storage permissions removed from user roles.
• If external parties need data: shares or gateways, never storage credentials or presigned URLs with long expiry.
• Small teams: a managed lake service with one catalog and one engine is usually safer than assembling open components. Large estates should standardise on one catalog per domain and make "no standing storage access for humans" a platform rule.

Catalogs: verify enforcement, not just configuration

Catalog products often ship in a compatibility mode that leaves enforcement off, so a governance programme can be fully configured and still not control anything.

• Open-source Unity Catalog: privileges are checked only when server.authorization=enable is set in server.properties; otherwise every request is allowed. With authorisation on, the first start creates an admin user and writes its token to etc/conf/token.txt, which must be removed or protected immediately. Access needs USE CATALOG and USE SCHEMA on parents plus SELECT or MODIFY on the table; non-owner privileges on a catalog or schema are inherited by everything below it, so grant at the lowest level that works.
• AWS Lake Formation: a resource that still grants permissions to the IAMAllowedPrincipals group is treated as an AWS Glue resource and Lake Formation permissions are not enforced. In hybrid access mode, only principals that have been opted in need Lake Formation permissions; everyone else continues on IAM alone. Track the list of opted-in principals and the resources still in hybrid mode as migration debt with an end date.
• Hive Metastore: the metastore itself usually has no fine-grained authorisation; protect it with Kerberos or network isolation and enforce access in the engines (for example Ranger), as described in the Hadoop Security Guide.
• Iceberg REST catalogs: vended credentials should be scoped to the table location and short-lived. Confirm in testing that a credential issued for one table cannot list or read another table's prefix, and that write credentials are not issued for read requests.

Treat catalog administrators and catalog service identities as privileged: whoever can grant on the catalog, or whose identity backs the storage credential, can read the whole lake. See the Service Account Security Guide and the Cloud Workload Identity Guide for the pipeline identities that write to it.

Encryption: storage, file and column level

Parquet modular encryption

The Apache Parquet encryption specification encrypts each module of a file (pages, page headers, column indexes, bloom filters and the footer) separately, so engines can still prune columns and push down predicates. Its goals include client-side encryption in which the storage server never sees plaintext data, metadata or keys; different keys for different columns and for the footer; and partial encryption of only the sensitive columns. Two algorithms are defined: AES_GCM_V1, which authenticates every module, and AES_GCM_CTR_V1, which encrypts pages with CTR for speed, where the specification notes that tampering with page data might go unnoticed. The footer can be encrypted, or left in plaintext and signed so that legacy readers can still read unencrypted columns. Keys are wrapped by a KMS (envelope encryption), so the key access policy becomes a second, independent access control on top of storage permissions.

Use column encryption when a column must be protected even from people who legitimately administer storage, for example national identifiers or card data held in a shared lake. Prefer AES_GCM_V1 where integrity matters more than throughput.

Key custody

• Separate keys per zone and per sensitivity: a single key for the whole lake means a single key policy decides everything.
• Keep keys away from the data: never store wrapping keys, KMS credentials or customer-provided keys alongside the files they protect. The GoTo breach shows what happens when encrypted backups are stolen together with a key.
• Block attacker-supplied keys: where storage allows client-provided keys (S3 SSE-C), deny them by policy unless needed. The Codefinger S3 ransomware campaign used stolen long-term keys to re-encrypt bucket contents with SSE-C keys only the attacker held.

See the Cryptographic Key Management Guide for rotation and KMS administration.

Governance: classification, lineage, retention and erasure

• Classify at ingestion: scan and tag data as it lands in the raw zone, not months later. Tags should drive access policy (tag-based access control) so new tables inherit protection automatically. See the Data Classification Guide.
• Minimise before promotion: tokenise or drop direct identifiers before data moves from raw to curated, so the widely-shared zones hold less sensitive data. See the Tokenisation and Data Masking Guide.
• Lineage: record which pipelines produced which tables, so that an access decision or erasure request can be traced to every derived copy, including feature stores and model training sets.
• Ownership: every table needs a named data owner who approves access and reviews it periodically.

Snapshots, time travel and the right to erasure

Open table formats keep history. In Apache Iceberg each write creates a new snapshot, and data files referenced only by old snapshots remain until snapshots are expired with expireSnapshots and orphan files are removed; metadata files can be cleaned with write.metadata.delete-after-commit.enabled. In Delta Lake, VACUUM removes files no longer referenced by the table, with a default retention threshold of 7 days, and it never runs automatically; running it with a shorter retention requires disabling the spark.databricks.delta.retentionDurationCheck.enabled safety check, and removes time travel to older versions.

The consequence for privacy: a DELETE statement that removes a person's rows does not remove their data from storage. Until snapshots are expired and files vacuumed, the rows are still readable through time travel or by reading old files directly. An erasure process under GDPR Article 17 should therefore include the delete, snapshot expiry or vacuum within a defined window, and the same in every derived table. The Data Retention and Secure Deletion Guide covers retention schedules.

The same history is a resilience asset: snapshots allow rollback after a bad or malicious write. Balance the two by keeping short history on tables with personal data and longer history, plus immutable backups outside the lake, on business-critical tables. See the Backup Security and Immutable Backup Guide.

Self-hosted and hybrid lakes

On-premises lakes built on HDFS or S3-compatible object stores carry the patching and exposure burden themselves. MinIO's advisory for CVE-2023-28432, which is in CISA's Known Exploited Vulnerabilities Catalog, describes how distributed deployments returned environment variables, including MINIO_SECRET_KEY and MINIO_ROOT_PASSWORD, to requesters; the fix was in RELEASE.2023-03-20T20-16-18Z and no workaround was offered. Lessons that apply to any self-hosted store:

• Never use the root or administrator credential for applications; create scoped users or policies per pipeline and keep the root credential in a vault.
• Keep admin and console endpoints off user networks and the internet.
• Track the store and its catalog against advisories and the KEV catalog as part of vulnerability management.
• For HDFS-based lakes, Kerberos, Ranger and encryption zones provide the equivalent controls; see the Hadoop Security Guide.

Implementation roadmap

1. Map access paths: for each zone, list every identity with storage permissions, every catalog, every engine and every share. This is usually where the surprise is.
2. Turn on logging first: object-level data access logs, catalog audit logs and engine query logs, so the following steps can be validated.
3. Choose the access model per zone using the decision guidance above, and write it down as a platform standard.
4. Pilot on one curated zone: enable catalog enforcement (for example remove IAMAllowedPrincipals grants or enable Unity Catalog authorisation) for a small group of tables, run in parallel with the old access for two to four weeks, and compare logs for denied access that should have been allowed.
5. Remove standing storage access from human and tool roles in the pilot zone. Rollback is re-attaching the previous storage policy, so keep it versioned.
6. Extend zone by zone, raw zone last but with the strictest outcome (engine or pipeline identities only).
7. Add encryption and erasure controls: per-zone keys, column encryption for the highest-sensitivity fields, and scheduled snapshot expiry and vacuum.
8. Operate: quarterly access reviews by data owners, share reviews, and the metrics below.

Detection and measurement

• Storage reads by identities that normally access data only through the catalog or engine (a bypass signal), mapped to MITRE ATT&CK T1530 Data from Cloud Storage.
• Listing of many prefixes or buckets by one identity (T1619 Cloud Storage Object Discovery).
• Large copies to accounts or buckets outside the organisation (T1537 Transfer Data to Cloud Account).
• Bulk overwrite, delete, lifecycle or snapshot-expiry changes outside maintenance windows (T1485 Data Destruction, T1486 Data Encrypted for Impact).
• Catalog grant changes, new external locations or storage credentials, and new shares.
• Use of credentials vended to one client from a different network location.

Framework mapping

Common failure patterns

• Governance configured, enforcement off: catalog grants carefully designed while IAMAllowedPrincipals grants or a disabled authorisation flag mean nobody is actually restricted.
• Vended credentials alongside standing access: the catalog issues scoped tokens, but analysts' roles can still read the whole bucket, so the catalog is advisory.
• Long-lived keys in pipelines and notebooks: access keys or service principal secrets in job configuration, the pattern behind the Codefinger S3 campaign. See the Secrets Management Guide.
• Erasure that stops at DELETE: rows removed from the current table version but still present in snapshots, old files and derived tables.
• An ungoverned raw zone: masking and tokenisation applied in curated tables while the raw landing area, holding everything, is readable by every data engineer.
• Self-hosted stores run on the root credential and left unpatched, so one disclosure bug yields full control.

Practitioner checklist

• Document the access model for every zone and confirm no zone mixes storage-direct user access with catalog governance.
• Verify catalog enforcement is on: no IAMAllowedPrincipals grants on governed resources, server.authorization=enable on Unity Catalog, Ranger plugins active in engines.
• Remove standing storage permissions from human and BI roles where a catalog vends credentials.
• Test that a vended credential for one table cannot read another table's location.
• Use separate encryption keys per zone, deny client-provided keys where not needed, and apply Parquet column encryption to the most sensitive fields.
• Tag personal data at ingestion and drive access from tags.
• Include snapshot expiry and vacuum in every erasure and retention process.
• Replace long-lived storage keys in pipelines with federated, short-lived credentials.
• Patch self-hosted object stores and catalogs against advisories and the CISA KEV catalog; keep root credentials vaulted.
• Log storage data events, catalog changes and engine queries, and alert on catalog bypass and bulk destructive actions.

Standards and references

• Apache Iceberg: REST Catalog OpenAPI specification
• Apache Iceberg: Maintenance
• Apache Parquet: Parquet Modular Encryption
• Delta Lake: Table utility commands (VACUUM)
• Unity Catalog: Authentication and authorization
• AWS: Lake Formation hybrid access mode
• MinIO: Information disclosure in cluster deployment (CVE-2023-28432)
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: Enterprise Data Lake Security Guide · Hadoop Security Guide · Data Security Guide · Databricks Security Guide · ETL and ELT Pipeline Security Guide