Why do organisations struggle to answer basic questions about their data environment?

Why This Matters for Security Teams

Basic data questions expose whether governance is actually operating or only documented. If teams cannot answer what data exists, where it lives, who owns it, and whether it is fit for use, then risk decisions are being made on partial evidence. NIST’s NIST SP 800-53 Rev 5 Security and Privacy Controls treats inventory, accountability, and configuration control as foundational, not optional.

This gap matters because fragmented data estates create blind spots across compliance, retention, access control, and incident response. NHIMG research shows that only 5.7% of organisations have full visibility into their service accounts, a useful reminder that visibility failures are rarely isolated to one asset class; they usually reflect a broader governance weakness documented in the Ultimate Guide to NHIs — Key Research and Survey Results. When metadata is inconsistent, stewardship is unclear, and ownership stops at team boundaries, data discovery becomes an exercise in manual reconciliation instead of operational control. In practice, many security teams encounter missing lineage only after a data exposure, failed audit, or access dispute has already occurred, rather than through intentional governance design.

How It Works in Practice

The practical problem is not just that data exists in many places. It is that organisations often manage it through separate tooling, conflicting naming conventions, and partial records that do not connect business context to technical location. A catalogue helps, but only if it is backed by shared definitions, stewardship, and policy enforcement. NIST guidance on inventory, accountability, and least privilege is relevant here, because the same control logic that protects systems also helps answer basic data questions when applied consistently.

Effective governance usually requires four linked capabilities:

• A single catalogue or register that records datasets, owners, systems, classifications, and retention rules.
• Clear stewardship so business ownership is explicit, not implied by platform administration.
• Metadata standards that make lineage, sensitivity, and lawful basis searchable across tools.
• Policy checks that validate access, sharing, and retention against the current record, not tribal knowledge.

That is why data teams often pair cataloguing with control frameworks and operational evidence from sources like the Ultimate Guide to NHIs — Key Research and Survey Results when they need to show how fragmented identity and secrets management creates the same visibility problem across machine-led data flows. The goal is not perfect centralisation, which is rarely realistic, but reliable attribution: knowing who can answer for a dataset, what systems touch it, and whether the current state matches policy. This aligns with NIST control families for configuration, access control, and auditability, and it is reinforced by NIST SP 800-53 Rev 5 Security and Privacy Controls.

These controls tend to break down when mergers, shadow IT, or unmanaged analytics platforms create parallel data estates that the catalogue never captures.

Common Variations and Edge Cases

Tighter governance often increases operational overhead, so organisations must balance visibility gains against the cost of continuous stewardship. That tradeoff is especially visible in fast-moving environments such as product analytics, self-service BI, and federated data mesh models, where teams want autonomy but still need answerable records.

There is no universal standard for how much metadata must be mandatory before a dataset is considered governed. Current guidance suggests starting with the fields needed for accountability and risk decisions: owner, source, classification, retention, and access constraints. Highly regulated environments may require lineage, legal basis, and regional residency as well. Less mature organisations often overbuild the catalogue before agreeing on ownership, which produces a polished inventory that no one trusts.

Another edge case is semi-structured and ephemeral data, where logs, exports, and temporary analysis sets move quickly between tools. In those environments, the best practice is evolving toward automated discovery and policy-as-code rather than manual documentation alone. The most reliable programmes treat governance as an operating model, not a one-time data cleanup. Where that model is missing, the same pattern appears across domains: teams can point to systems, but they cannot confidently explain what the data means or who is responsible for it.

Related resources from NHI Mgmt Group

• What breaks when organisations cannot answer basic questions about data lineage and permitted use?
• How should organisations answer critical data governance questions before expanding analytics and AI use cases?
• How should organisations build a data governance model that can answer who owns data, who can use it, and why it matters in business context?
• What do organisations get wrong about proving the impact of data governance programs?