TL;DR: Traditional DLP cannot inspect Parquet, Avro, ORC, or Feather at the scale and structure used in modern data lakes, leaving PII and credentials hidden in shared analytics stores, according to Sentra. The governance issue is not file volume alone, but the mismatch between data engineering access patterns and control models built for emails and documents.
At a glance
What this is: This article argues that modern data lake file formats create a DLP blind spot because conventional scanners cannot understand columnar and schema-embedded storage.
Why it matters: IAM and DSPM teams need visibility into where sensitive data lands, because broad read access to lakehouse stores can turn masked source data into exposed analytics data.
Context
Data lake security fails when controls assume files are simple text objects instead of structured containers with embedded schemas and columnar layouts. In lakehouse environments, the problem is not just storage scale but the mismatch between data engineering pipelines and inspection tools designed for emails, PDFs, and spreadsheets. That creates a governance gap for both data security and identity-driven access to shared analytics stores.
The identity angle is real because lake access is usually granted through IAM roles, not per-record controls. When masked production data is copied into object storage and the copy stays intact, the question becomes who can read the file, what the file contains, and whether the organisation can prove that sensitive fields are actually controlled. That is a data governance problem with direct access-control consequences.
Key questions
Q: Where does DLP fail in modern data lake architectures?
A: DLP fails when it is built for document-style inspection but asked to parse columnar or schema-embedded lake formats. Parquet, Avro, ORC, and Feather can hide sensitive fields from legacy scanners, so the control loses sight of data after it is copied into object storage. The fix is format-aware discovery tied to the actual storage model.
Q: Why do shared data lake permissions increase exposure risk?
A: Shared lake permissions increase exposure risk because they expand the number of identities and workloads that can reach intact sensitive data. If masking fails during ETL, broad read access turns analytics infrastructure into a high-blast-radius store. Access control is still necessary, but it cannot substitute for visibility into what the files contain.
Q: What do security teams get wrong about data lake scanning?
A: Teams often assume that scanning a storage bucket once is enough, when the real problem is field-level content hidden inside changing file formats and pipeline outputs. They also overestimate the protection provided by source-system masking if the copied data lands intact in the lake. Effective scanning has to follow the data structure, not just the location.
Q: What happens when analytics teams copy production data into a lake without reclassification?
A: Sensitive records can move from tightly governed systems into shared analytics stores with their original fields still intact. At that point, the risk is not just leakage, but uncontrolled reuse across BI, engineering, and machine learning workflows. Governance has to travel with the copy, or the copy becomes the new source of exposure.
Technical breakdown
Why traditional DLP misses columnar data lake formats
Traditional DLP is usually built to inspect linear text in documents, messages, or spreadsheets. Parquet and ORC store data column by column, while Avro embeds schema in the file itself and Feather is optimised for fast interchange rather than human-readable content. A scanner that does not understand those structures cannot classify sensitive fields accurately, so it may miss PII even when the data is plainly present. The issue is structural, not just a tuning problem.
Practical implication: DLP coverage for data lakes must be format-aware, not limited to legacy file inspection rules.
Why schema-aware scanning changes data lake classification
Schema-aware scanning reads the file structure before it inspects the content, which allows classification at the column or field level. That matters because a large lake file may contain a single sensitive column alongside many benign metrics. It also helps distinguish semantically different fields with similar numeric patterns, such as an SSN field versus a ZIP code field. In practice, the goal is not to flag an entire lake bucket as risky, but to identify exactly where sensitive data lives.
Practical implication: organisations should classify sensitive data at field level so remediation can target the exact columns that matter.
How IAM permissions amplify a data lake blind spot
Data lake exposure is rarely caused by storage alone. The risk grows when broad IAM roles grant read access to shared analytics buckets, notebooks, and downstream tools. If copied production data still contains unmasked personal data, any user or workload with those permissions can reach it, even if the original source system was better controlled. That turns access governance into a multiplier for data visibility problems. The control failure is not only detection, but also the decision to centralise data without equivalent inspection and entitlement boundaries.
Practical implication: entitlement reviews must be paired with data discovery so broad read access does not become broad data exposure.
Threat narrative
Attacker objective: The practical objective is unauthorised discovery and reuse of sensitive data that was assumed to be protected once it entered the lake.
- Entry occurs when production data is copied into lakehouse storage as Parquet, Avro, ORC, or Feather during ETL or analytics preparation.
- Sensitive fields remain intact because masking or anonymisation does not survive the pipeline, and conventional DLP fails to inspect the embedded structure.
- Broad IAM-based read permissions allow analysts, engineers, or downstream workloads to access the copied records at scale.
- The result is hidden exposure of PII, credentials, or regulated data across shared analytics stores and data lake buckets.
Breaches seen in the wild
- DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
- Firebase misconfiguration exposure 2024: Missing Firebase security rules on 916 websites exposed 125 million user records and 19.87 million plaintext passwords; a quarter were fixed.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Data lake DLP blind spots are really governance blind spots. The core problem is not that data exists in object storage, but that security teams lose field-level visibility once data is transformed into lake-native formats. When inspection stops at the file boundary, masking assumptions, classification labels, and access controls all become weaker than the pipeline that created the copy. Practitioners should treat format-aware discovery as a governance prerequisite, not an enhancement.
Schema-aware visibility is the named control gap here: the enterprise cannot govern what it cannot parse. Columnar and embedded-schema formats break legacy inspection assumptions because the sensitive unit is no longer the document, it is the field. That means DLP, DSPM, and access governance need to converge on the same underlying inventory of data locations and types. Practitioners should align scanning depth with the actual storage model, not the legacy file model.
IAM creates scale, not safety, when data copies retain sensitive content. Shared lake permissions are efficient for analytics, but they also expand blast radius when copied records are still live and readable. The governance mistake is treating the copy as less sensitive than the source because it sits in a different system. Practitioners should assume lake access is production-grade access whenever the payload is still intact.
Data democratization without inspection is a control design failure. The article shows a familiar pattern in modern security: access is widened first, then visibility is expected to catch up later. In lakehouse environments that sequence is backwards because the number of files, formats, and consumers grows faster than manual review can keep up. Practitioners should design for discovery at ingestion, not rescue after exposure.
AI and analytics pipelines inherit the same data governance debt. Training datasets and analytical stores increasingly share the same lake formats, which means sensitive data can flow from operational systems into model inputs without a clear governance checkpoint. That is an identity and data-security intersection because the same role-based access that supports analytics can also expose regulated data to downstream machine learning use. Practitioners should govern lake data as reusable business data, not as a temporary technical artifact.
From our research library:
- Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
- Read next: NHI Lifecycle Management Guide
What this signals
Lakehouse programmes tend to fail at the point where access and discovery are treated as separate controls. Once sensitive data has been copied into Parquet, Avro, ORC, or Feather files, the organisation needs visibility at the file and field level, not just entitlement governance at the bucket level.
Schema-aware discovery gap: the control boundary has shifted from the application to the data lake itself, and that means data security teams need inventory, classification, and access review to operate on the same dataset. The Ultimate Guide to NHIs shows only 5.7% of organisations have full visibility into their service accounts, a reminder that control gaps often start with incomplete inventory rather than a single failed policy.
For practitioners, the immediate implication is that lake access reviews should be paired with scanning coverage checks for every format used in ETL and analytics. If the control cannot parse the file structure, it cannot support defensible governance over the records inside it.
For practitioners
- Implement format-aware DLP for lakehouse storage Prioritise tools that can read Parquet, Avro, ORC, and Feather natively, then validate that they inspect column-level content instead of treating each file as an opaque blob.
- Classify sensitive fields at ingestion Tag and detect sensitive columns as data enters the lake so masking, classification, and downstream policy decisions do not depend on later manual review.
- Review broad IAM access to shared analytics stores Check which roles can read lake buckets, notebooks, and query layers, then narrow access where those identities do not need intact personal or regulated data.
- Validate masking survives ETL and replication Test whether source-system anonymisation remains effective after pipelines land data in object storage, especially when production records are copied for analytics or ML training.
Key takeaways
- Modern data lakes expose a governance gap because legacy DLP tools cannot inspect the formats where sensitive fields now live.
- Visibility must move from bucket-level review to field-level discovery if organisations want to control reused production data in analytics pipelines.
- IAM permissions can widen exposure quickly when masked data is copied intact into shared lakehouse stores, so discovery and access control have to operate together.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 addresses the attack surface, NIST CSF 2.0, CIS Controls v8 and CSA Cloud Controls Matrix set the technical controls, and ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-10 — Human Use of NHI | Lake access is governed through human-assigned IAM roles that can widen visibility into sensitive shared data. |
| Recommendation — Review who can use human-granted access paths to reach lake data and remove standing permissions that exceed need. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | The article centres on broad read access to shared analytics stores and the need to align entitlements with data sensitivity. |
| Recommendation — Map lakehouse read access to PR.AA-05 and reduce entitlements that expose intact sensitive records. | ||
| CIS Controls v8 | CIS-5 — Account Management | Shared data lake access depends on the lifecycle and scope of accounts and roles granted into analytics environments. |
| Recommendation — Use CIS-5 to review which accounts and roles can read lake stores and retire unnecessary access paths. | ||
| CSA Cloud Controls Matrix | IAM — Identity and Access Management | Cloud lake buckets and analytics platforms are governed through IAM, which determines who can read copied data. |
| Recommendation — Apply CCM IAM controls to constrain who can read shared lake data and to validate least-privilege access. | ||
| ISO/IEC 27001:2022 | A.5.12 — Classification of information | The article is about failing to identify sensitive data once it is stored in new file formats and shared repositories. |
| Recommendation — Use A.5.12 to classify lake data by sensitivity before broad analytics access is granted. | ||
Key terms
- Columnar Database: A columnar database stores data by column rather than by row, which makes it efficient for analytics over large datasets. This layout improves compression and accelerates queries that scan a subset of fields across many records. It is commonly used for log analysis, reporting, and other read-heavy workloads.
- Schema-Aware Scanning: Schema-aware scanning is inspection that understands how a file is structured before it looks for sensitive content. In data lakes, that means reading columns, embedded schemas, and nested types so classification can work on Parquet, Avro, ORC, and similar formats rather than treating them as opaque blobs.
- Data lakehouse: A data lakehouse is a cloud data architecture that combines the scale of object storage with the management features of a warehouse-like table layer. It supports AI and analytics workloads, but it also increases the importance of metadata integrity, recovery testing, and lifecycle governance.
- Field-Level Classification: The process of identifying and labeling sensitive data at the column or attribute level rather than treating an entire file as a single unit. This gives security teams a more precise picture of where regulated or confidential information actually resides.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the NHI Foundation Level course, the industry's only accredited NHI security programme, if your role touches identity, access, or machine credential governance.
Published by the NHIMG editorial team on August 19, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org