Enterprise Data Lake Security Guide: Access, Encryption and Governance

An enterprise data lake concentrates the organisation's most sensitive data in object storage that is reachable through more paths than any single database ever was. Analysts query it through a catalog, pipelines write to it with service identities, notebooks and BI tools read it with their own credentials, and the storage account or bucket underneath still answers to its own keys and policies. Security fails when those layers disagree. This guide covers data lakes and lakehouses built on Amazon S3, Azure Data Lake Storage Gen2 and Google Cloud Storage, with catalog-level access control through AWS Lake Formation, BigLake and similar layers. It is written for cloud security engineers, data platform owners and architects. Security log lakes used for detection are a different problem, covered in the Security Data Lake Guide.

Key takeaways

• Close the storage bypass first. Catalog permissions mean little if a principal can still read the bucket directly with an access key, a Shared Key or a broad object-read role.
• Govern with tags, not grant lists. Attribute-based grants (Lake Formation LF-Tags, Unity Catalog or BigLake policies) scale with principals plus resources, where named grants scale with principals times resources.
• Pipelines are the biggest identity population. Ingestion jobs, ETL roles, connector service accounts and BI integrations hold standing access to whole zones and need the same lifecycle discipline as human admins.
• Encryption is a control plane decision. Who can use or change the key, and which encryption modes are allowed, matters more than whether data is encrypted, as SSE-C ransomware against S3 has shown.
• Log data access, not just configuration. Object-level data events and catalog audit logs are off or partial by default on most platforms and are the only way to answer "who read this table".

Data lake architecture and where the trust boundaries sit

Most enterprise lakes follow the same layered pattern. Each layer has its own authorisation system, and an attacker or careless user only needs the weakest one.

Zones (often called raw, curated and consumption, or bronze, silver and gold) are a useful boundary. Raw zones tend to hold the least filtered personal data and the fewest controls. Treat the raw zone as the most sensitive, not the least, and put zones in separate buckets, containers or accounts so storage-level policy can differ between them.

Access control: catalog, storage and the bypass problem

AWS: Lake Formation over S3

• Two checks, both required: a request to Lake Formation-managed data succeeds only if it passes both the IAM policy check and the Lake Formation permission check. Lake Formation separates metadata (Data Catalog) permissions, data access permissions on the S3 data, and data location permissions that control who may create tables pointing at a given S3 path.
• The default is IAM only: for backward compatibility with AWS Glue, Lake Formation grants the Super permission to the IAMAllowedPrincipals group on existing catalog resources and enables "Use only IAM access control" for new ones. Until that changes, Lake Formation grants have no effect. Switch new resources by setting CreateDatabaseDefaultPermissions and CreateTableDefaultPermissions to empty lists in PutDataLakeSettings, then upgrade existing databases and tables.
• Tag-based access control: LF-TBAC attaches LF-Tags such as classification=restricted to databases, tables and columns, and grants principals permissions on tag values. Tables inherit tags from their database and columns from their table. AWS describes the scaling difference directly: named grants need n(principals) x n(resources), tag grants need n(principals) + n(resources). After moving to LF-TBAC, AWS recommends revoking the old named-resource grants once access is verified. LF-Tags are not IAM tags; they are not interchangeable.
• Delegation: data lake administrators grant permissions, can appoint LF-Tag creators, and can let grantees pass permissions on with the grant option. Keep the grant option rare.

See the AWS IAM Policies and Roles Guide for the IAM half of the check.

Azure: ADLS Gen2

• Evaluation order: Azure evaluates RBAC role assignments first, then ABAC conditions on those assignments, and only then POSIX ACLs. An ACL cannot restrict access that a role assignment already grants, so a broad Storage Blob Data Reader assignment at account level makes every directory ACL irrelevant.
• Keys bypass everything: Shared Key, account SAS and service SAS carry no Microsoft Entra identity, so RBAC, ABAC and ACLs do not apply. Shared Key gives what Microsoft calls super-user access, including changing owners and ACLs. Management roles such as Owner, Contributor and Storage Account Contributor do not grant data access directly, but can list the account keys and therefore read everything.
• Disable Shared Key: setting AllowSharedKeyAccess to false makes Shared Key requests fail with 403; account and service SAS stop working, user delegation SAS (signed with Entra credentials) keeps working. Microsoft notes that Conditional Access protection for the account requires this. Detect remaining use first with StorageBlobLogs | where AuthenticationType in ("AccountKey", "SAS"), then assign the built-in policy "Storage accounts should prevent shared key access" in Audit, then Deny. Check Azure Files and Cloud Shell dependencies before switching.
• ACL hygiene: ACLs are limited to 32 entries per file or directory (access and default ACLs each), so assign Entra security groups, never individual users or service principals, and keep a principal's group memberships under 200 to avoid token group-claim problems.

See the Microsoft Azure Security Guide.

Google Cloud: Cloud Storage and BigLake

• Uniform bucket-level access: UBLA disables object ACLs so only IAM grants access. It is required for IAM Conditions, managed folders and Workforce or Workload Identity Federation access, and cannot be disabled once it has been on for 90 consecutive days. Check the storage.googleapis.com/authz/acl_operations_count metric before enabling it to find clients that still rely on object ACLs.
• Access delegation: BigLake tables read Cloud Storage, Amazon S3 or Blob Storage through a connection tied to a service account, so analysts need access to the table, not the bucket. Google's own documentation warns that giving analysts direct object-read roles lets them bypass table-level controls. BigLake supports row-level and column-level security, and dynamic data masking for Cloud Storage-based tables.

See the Google Cloud IAM and Service Accounts Guide.

The bypass problem in one table

Identity for pipelines, connectors and consumers

In most lakes, non-human identities outnumber analysts and hold broader access. Ingestion roles write to raw zones, transformation jobs read raw and write curated, BI connectors read consumption tables on behalf of hundreds of users, and external sharing creates cross-account principals.

• One identity per pipeline: give each ingestion source and each transformation job its own role or service account, scoped to the zones it touches. A shared "etl-role" with read and write on every bucket turns any compromised job into a full lake compromise.
• No static keys where federation exists: use IAM roles, managed identities, Google service account impersonation or workload identity federation instead of access keys, storage account keys and service account key files. See the Workload Identity Federation Guide.
• Separate write from grant: the identity that loads data should not be able to change catalog permissions, bucket policies or encryption settings.
• BI and reverse ETL connectors: these often authenticate with a single service account and re-share data to many users. Know which tables each connector reads and whether the downstream tool enforces its own row-level rules.
• Cross-account and external sharing: inventory every share, its receiving account and its expiry. Lake Formation, Unity Catalog and BigQuery all support sharing that never appears in the source bucket policy.

The 2024 Snowflake customer breaches show what happens when data platform credentials are protected by password alone: infostealer-harvested credentials without MFA or network allow lists gave UNC5537 access to around 165 organisations' data. See the Service Account Security Guide.

Encryption and key management

• Deny SSE-C unless needed: AWS now disables SSE-C by default for new general purpose buckets and advises keeping it off unless a workload needs it. In the Codefinger S3 ransomware campaign, stolen AWS keys were used to re-encrypt objects with SSE-C keys the attacker held, and a lifecycle rule set a seven-day deletion deadline. Check existing buckets as well as new ones.
• Use the key policy as a boundary: grant kms:Decrypt (or the Azure and Google equivalents) only to the roles that should read a zone. A principal with storage access but no key access then gets an error, not data.
• Enforce encryption in transit: deny S3 requests where aws:SecureTransport is false and use the s3-bucket-ssl-requests-only AWS Config rule; Azure and Google enforce TLS for their storage endpoints, but check minimum TLS version settings on Azure storage accounts.

See the Cryptographic Key Management Guide and the Encryption Guide.

Governance: classification, tagging and data lifecycle

• Classify at ingestion: sensitive data discovery (for example Amazon Macie, which finds PII, financial data and credentials in S3 and reports object, type and count) should run on raw zones, where unreviewed data lands. Feed results into catalog tags rather than a separate spreadsheet. See the Data Classification Guide.
• Make tags drive access: one tag vocabulary (for example classification, domain, residency) applied in the catalog and used directly by LF-TBAC, ABAC conditions or policy tags.
• Column and row controls in the catalog: masking and row filters belong in the catalog layer so every engine enforces them, not in each BI tool.
• Retention and deletion: the lake is where deleted records live on. Apply lifecycle rules per zone, account for open table format snapshots and time travel, and confirm that erasure requests reach raw copies and backups.
• Integrity and recovery: versioning and S3 Object Lock (WORM) protect against deletion and tampering; restrict who can change lifecycle and replication rules, because those are destructive actions too.
• Lineage: lineage from source to report is what lets you answer which downstream tables inherit a sensitive column.

For broader data security programme design, see the Data Security Guide.

Network exposure and public access

• Block public access at the top: enable S3 Block Public Access at organisation level, Google public access prevention, and disable anonymous blob access on Azure storage accounts. AWS recommends the Config rules s3-bucket-public-read-prohibited and s3-bucket-public-write-prohibited as detective controls.
• Disable ACLs: S3 Object Ownership "bucket owner enforced" disables ACLs; after that, PUT requests with custom ACLs fail with AccessControlListNotSupported. UBLA does the same on Cloud Storage.
• Private paths: reach storage through VPC endpoints or private endpoints and restrict bucket policies to them (on AWS with the aws:SourceVpce condition key). Test deny statements carefully: they also block console and pipeline access from outside the VPC.
• External access review: IAM Access Analyzer for S3 flags buckets shared with the internet or other accounts, including outside the organisation. Record an owner for each intended share.

Detection and monitoring

Map detections to MITRE ATT&CK: T1530 Data from Cloud Storage, T1078.004 Cloud Accounts, T1537 Transfer Data to Cloud Account, T1485 Data Destruction and T1486 Data Encrypted for Impact. Send lake logs to your detection platform; the Detection Engineering Guide covers rule design.

Metrics that show the controls are working:

• Percentage of lake storage accounts and buckets with key-based or ACL-based access disabled (target 100 percent).
• Number of principals with direct object read on curated and consumption zones outside the catalog's service identities (target trending to zero).
• Percentage of tables with a classification tag, and percentage of grants made by tag rather than by name.
• Percentage of lake buckets and containers with data access logging enabled and retained.
• Median age of pipeline credentials, and count of static access keys or account keys in use.
• Open external-access findings older than 30 days.

Implementation roadmap

1. Inventory: list every bucket, container and catalog that holds lake data, their zones, owners and the principals (human and non-human) with storage-level access. Use Access Analyzer, Azure Resource Graph (for allowSharedKeyAccess) and IAM policy analysis.
2. Turn on logging before changing access: enable data access logs for at least the raw and restricted zones, so you can see who will break when you tighten controls.
3. Remove public and legacy paths: organisation-level public access blocks, ACLs disabled, SSE-C denied. These rarely break legitimate lake workloads.
4. Audit-mode the key bypass: on Azure, run the Shared Key policy in Audit and migrate callers to Entra identities; on AWS, list roles with direct S3 read to lake paths; on Google, list object-read bindings on BigLake buckets.
5. Enforce catalog control on a pilot domain: pick one data domain, switch Lake Formation defaults (or equivalent), apply tags and tag-based grants, and remove direct storage read for its consumers. Keep the previous IAM policies versioned so you can restore them if a critical pipeline fails.
6. Roll out by zone: restricted and raw zones next, then the rest. Switch Azure Policy to Deny and set AllowSharedKeyAccess to false once logs show no remaining key traffic.
7. Rebuild pipeline identities: one role per job, federated credentials, write separated from grant, keys retired.
8. Operate: quarterly access reviews per domain, monitoring of the metrics above, and a review whenever a new engine or sharing feature is enabled.

Small versus large organisations: a small team with one cloud and a handful of pipelines can often keep storage IAM as the main control, provided zones are separate buckets, keys are disabled and logs are on. Catalog-level tag-based access pays off once there are many domains, many consumers or a need for column and row controls; it adds a second policy system to operate, so adopt it deliberately rather than half-way, where both systems grant access.

Common failure patterns

• Catalog enabled, bypass left open: Lake Formation or a lakehouse catalog is deployed, but the old IAM roles with bucket read remain, or IAMAllowedPrincipals still holds Super. Reports show fine-grained control that does not exist.
• Account keys in connection strings: ADLS keys or account SAS embedded in notebooks, pipeline configs or BI tools, giving super-user access with no identity in the logs.
• Destructive actions treated as configuration: lifecycle rules, replication changes and encryption settings not alerted on. In the Codefinger campaign these became the ransom mechanism.
• Password-only access to a data platform: service and user logins with no MFA or network restriction, as in the Snowflake customer breaches.
• Raw zone as a free-for-all: engineers granted broad access to raw data "temporarily", which holds the least-filtered personal data in the organisation.
• No data events: only management events logged, so a bulk download leaves no trace and breach scoping relies on guesswork.

Framework mapping

Practitioner checklist

• Block public access at organisation level and disable object ACLs (S3 bucket owner enforced, GCS UBLA).
• Set AllowSharedKeyAccess to false on ADLS accounts after migrating callers, enforced with Azure Policy in Deny.
• Remove direct object read from consumer roles; only catalog service identities read curated and restricted zones.
• Switch Lake Formation off the IAM-only defaults (or the equivalent in your catalog) and revoke Super from IAMAllowedPrincipals on migrated resources.
• Grant access by classification tags, not by named table lists, and keep the grant option rare.
• Give each pipeline its own federated identity, scoped to its zones, with no rights to change policies or keys.
• Use customer-managed keys for restricted zones and deny SSE-C where it is not required.
• Enable data access logging on every lake bucket and container, and alert on policy, lifecycle, replication and SSE-C events.
• Run sensitive data discovery on raw zones and feed results into catalog tags.
• Review external shares and cross-account grants quarterly, each with a named owner and expiry.

Standards and references

• AWS: Lake Formation permissions
• AWS: Lake Formation tag-based access control
• AWS: Security best practices for Amazon S3
• Microsoft Learn: Access control model in Azure Data Lake Storage
• Microsoft Learn: Prevent Shared Key authorization for an Azure Storage account
• Google Cloud: Introduction to BigLake tables
• Google Cloud: Uniform bucket-level access
• MITRE ATT&CK: Data from Cloud Storage

Related NHI Mgmt Group resources: Security Data Lake Guide · Databricks Security Guide · Snowflake Security Guide · Data Security Guide · Cloud Security Guide