Dark Data vs Shadow Data: Unused Data vs Unmanaged Data

Dark data is data you hold and know about but never use; shadow data is data that exists outside the stores you manage, often a copy you do not know about at all. The two overlap, because a forgotten copy is usually also unused, but they call for different responses: dark data is mainly a retention and value decision, shadow data is mainly a discovery and access control problem. IBM's Cost of a Data Breach 2024 found that more than a third of breaches involved shadow data, which it described as data stored in unmanaged data sources. This guide separates the two, explains how pipelines, backups and AI workloads create both, looks at the human and non-human identities that can reach them, and sets out how to find them and decide what to keep or delete. It is written for data security, privacy and cloud teams. For the tooling itself, see the Data Security Posture Management (DSPM) guide.

Key takeaways

• Short answer: dark is unused, shadow is unmanaged. Dark data sits in a known, governed store but serves no current purpose. Shadow data sits somewhere your inventory, classification and access controls do not reach.
• Most shadow data starts as a legitimate copy. Exports, test datasets, analytics extracts, backups, snapshots and AI training sets are made for a reason, then outlive it and drift out of governance.
• Identities are the exposure path. Service accounts, pipeline roles, shared links and API keys that can read shadow copies are rarely reviewed, because the copy is not on anyone's access review list.
• Discovery must cover where data lands, not just where it should live. DSPM-style scanning of cloud storage, databases, SaaS and data platforms finds shadow copies; access and query logs show which data is dark.
• Deletion is the strongest control for both. Under storage limitation (UK GDPR Article 5(1)(e)) data kept without a current purpose is a compliance issue as well as a security one. Decide, delete, and extend deletion to backups and derived copies.

What each one is, in practice

Dark data: known but unused

IBM describes dark data as information that organisations accumulate but often never use for analytics or decision-making. It is collected in normal operations and then left: old log archives, call recordings, legacy CRM exports, IoT sensor history, closed-project shares, email archives. IBM cites Splunk's 2019 State of Dark Data survey, in which 60% of respondents said half or more of their organisation's data was dark. The defining feature is that the store is known and usually governed. Someone could find it, and access to it is controlled, but nobody has asked whether it should still exist.

Shadow data: outside managed stores

Shadow data is data held in places the security and data teams do not manage or do not know about. It may be heavily used. A spreadsheet extract an analyst refreshes daily from a production database into a personal cloud folder is shadow data that is very much alive. The defining feature is a governance gap: the location is missing from the data inventory, its classification is unknown, and access to it is not reviewed.

Where they overlap

The overlap is the riskiest quadrant: an unknown copy that nobody uses. A one-off migration snapshot, a test dataset copied from production two years ago, a backup of a departed employee's workstation in a storage account. Nobody will notice if it is stolen, and nobody needs it.

Side-by-side comparison

How copies are created: pipelines, backups and AI

Most shadow data and much dark data is a copy. Knowing which processes copy data tells you where to look.

AI deserves its own note. OWASP's LLM02:2025 Sensitive Information Disclosure covers sensitive data reaching model outputs, and training on uncontrolled copies is one route there. Embeddings and vector databases hold derived representations of source documents, so treat them with the source's classification, and record every dataset copied into an AI platform as a new data asset with an owner. See AI Data Retention and Deletion and Data Lineage vs Data Provenance.

Shadow data in depth: threat model, architecture and failure modes

Threat model: an attacker who obtains any credential with storage read access, or finds a public or link-shared object, collects data that no monitoring is watching. This maps to ATT&CK T1530 Data from Cloud Storage and T1213 Data from Information Repositories. Insiders and departing staff can take shadow copies with the same ease, because no data loss rule covers a location nobody classified.

Architecture in words: the governed platform (warehouse, lake, primary database) has classification, access reviews, logging and encryption policy. A copy leaves it through an export, a pipeline step or a snapshot, and lands in a storage account, bucket, SaaS folder or developer database that has none of those. The copy keeps the sensitivity of the source but loses the controls.

Failure modes:

• Storage containers created outside infrastructure as code, so they never receive standard tags, encryption or logging settings.
• Shared links and signed URLs granting broad, long-lived access. The Microsoft SAS token exposure is the textbook case: a link meant for public model files reached a whole storage account that also held workstation backups, for nearly three years.
• Production copies in non-production accounts where security tooling is not deployed.
• Data discovery scoped to "known" stores, which by definition excludes the shadow ones.

Controls: continuous discovery across every account, subscription and SaaS tenant (not a list of approved stores), default encryption and logging applied at the platform level, sharing link expiry, and a rule that any discovered sensitive copy gets an owner within a fixed time or is deleted.

Dark data in depth: threat model, architecture and failure modes

Threat model: dark data rarely causes the breach; it multiplies the impact. When an attacker reaches an archive share or log store, the years of unused records inside determine how many people must be notified. It also widens legal exposure: data held is data that may have to be searched for subject access requests and disclosure.

Architecture in words: a governed but neglected store, often on a cold or archive storage tier, sometimes on a legacy system kept running for one report. Access was granted years ago to groups and service accounts that still exist. Nobody reads the data, but the permissions remain.

Failure modes:

• No retention schedule, or one that exists on paper but has no technical enforcement.
• "Keep everything in case" decisions with no named purpose, which the ICO's storage limitation guidance says is not a justification.
• Moving data offline or to an archive tier and treating that as deletion. The ICO notes that offline data is still processed and still subject to subject access requests.
• Deleting live data but not its backups, which the ICO says deletion should extend to.

Controls: a retention schedule by data category, as the ICO expects, enforced with lifecycle rules and periodic review; owner attestation for stores with no reads; anonymisation where analytical value remains; and sanitisation of retired media following NIST SP 800-88 Rev. 2 (final, September 2025).

Where identity fits: who and what can reach each

Discovery usually starts with the data, but exposure depends on who can read it. Shadow and dark stores are dominated by non-human access.

Joining data findings with identity findings is where the value lies: a shadow copy readable by one dormant service account is a cleanup ticket; the same copy readable by a pipeline role whose key is in a public repository is an incident. See Data Access Governance Guide and Orphaned and Dormant Accounts.

Discovery: finding shadow copies and dark stores

1. Scope by account, not by store. Enumerate every cloud account, subscription, project and SaaS tenant, including non-production ones, then list every storage resource in each. Shadow data hides in the stores nobody added to the list.
2. Scan content. Use DSPM or cloud-native sensitive data discovery to sample and classify data in storage, databases and data platforms. The Data Classification vs Data Discovery guide covers detector design.
3. Match against the inventory. Any store with sensitive data that is not in the data inventory or record of processing is shadow. NIST CSF 2.0 ID.AM-07 calls for inventories of data and metadata for designated data types; this step tests yours.
4. Find copies by fingerprint. Compare schemas, column names, row counts or document hashes across stores to identify copies of known sensitive sources.
5. Measure use. Pull read activity from storage and query logs. Stores with no reads for 12 months or more are candidates for dark data review.
6. Map access. For each store, list the identities and links that can read it, and when each last did.
7. Assign owners. Every finding gets an owner, or a default owner for the account, with a deadline to decide.

Retention and deletion decisions

Under UK GDPR Article 5(1)(e), personal data must be kept in identifiable form for no longer than necessary. The ICO says organisations must set and justify retention periods themselves, document them, review them and erase or anonymise data that is no longer needed, with limited exceptions for archiving in the public interest, research and statistics under safeguards. The ICO notes this guidance is under review following the Data (Use and Access) Act, so check the current version.

Roll out deletion carefully: start with report-only lifecycle rules, move data to a restricted quarantine location with a short timer before final deletion, and keep a deletion log. See the Data Retention and Secure Deletion Guide and the Data Retention and Disposal Policy Template.

Decision guidance

• Lead with shadow if you are cloud-heavy, have many accounts or data science teams, or allow broad self-service exports. Exposure is the bigger risk.
• Lead with dark if you run large legacy estates, long-lived file shares or archives, or face many subject access or disclosure requests. Retention cost and breach impact dominate.
• Small organisations: a storage inventory per account, default lifecycle rules and a one-page retention schedule cover most of both problems without dedicated tooling.
• Large organisations: continuous DSPM discovery joined with identity data is usually needed; fund owner assignment as a process, not a project.

Worked scenario

A retailer's DSPM rollout scans all 140 cloud accounts rather than the 30 registered data platforms. It finds a bucket in a sandbox account holding a 2022 customer table export with card-free order history and addresses: shadow and dark at once. Access logs show no reads in 19 months; two identities can read it, an old pipeline role and a shared link with no expiry. The owner confirms no purpose, so the team revokes the link, removes the role's grant, deletes the bucket and its snapshots, and adds a lifecycle rule to all sandbox accounts. The same scan finds a vector store built from support tickets, used daily but unregistered: shadow, not dark. That one is registered, classified to match the ticket system, moved behind the governed access model and given a retention period aligned to the ticket data.

Detection and measurement

Log storage creation, sharing link creation, cross-account snapshot sharing and bulk reads, and alert when sensitive data appears in an unregistered store.

• Unregistered sensitive stores: stores with sensitive data that are not in the inventory. Target: trending to zero.
• Discovery coverage: share of cloud accounts and SaaS tenants scanned in the last 30 days.
• Stale sensitive data: volume of sensitive data with no reads in 12 months.
• Time to owner: median days from discovery to a named owner and decision.
• Deletion throughput: volume deleted or anonymised per quarter against the retention schedule.
• Non-human readers per sensitive store: count of service accounts, keys and links with read access, trending down.

Framework mapping

Common failure patterns

• Scanning only approved stores: discovery limited to the registered platforms cannot find shadow data by design.
• Treating archive as deletion: cold tiers and offline media still hold personal data, with the same obligations.
• Forgetting backups and derived copies: the source is deleted but snapshots, exports and embeddings remain.
• Findings without owners: a DSPM dashboard full of unassigned findings changes nothing.
• Ignoring machine readers: reviews cover human groups but not the pipeline roles, keys and links that can read the data.

Practitioner checklist

• Scope discovery to every cloud account, subscription and SaaS tenant, including sandbox and non-production.
• Reconcile discovered sensitive stores against the data inventory and record of processing.
• Pull read activity to flag stores with no access in 12 months.
• Map every service account, key and shared link with read access to each sensitive store.
• Register AI datasets, vector stores and prompt logs as data assets with owners and retention.
• Publish and technically enforce a retention schedule by data category, including backups.
• Require a deletion step in every migration and test data runbook.
• Track unregistered sensitive stores and time to owner monthly.

Standards and references

• ICO: Principle (e) Storage limitation
• NIST: SP 800-88 Rev. 2 Guidelines for Media Sanitization
• NIST: Cybersecurity Framework 2.0
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• OWASP: LLM02:2025 Sensitive Information Disclosure
• IBM: Cost of a Data Breach Report 2024 announcement
• IBM: What is dark data?

Related NHI Mgmt Group resources: DSPM vs DLP, Data Minimisation, Data Mapping and ROPA Guide, Backup Security and Immutable Backup Guide.