Why do unstructured data problems delay AI programmes?

Why This Matters for Security Teams

Unstructured data is not just a data quality issue. It is a delivery risk for AI programmes because models, retrieval layers, and agents all depend on clean context, traceable provenance, and consistent access to content. When that context is scattered across documents, tickets, chats, scans, and shared drives, teams spend time deciding what is authoritative before they can even test a use case. That delay is often magnified when secrets, personal data, or policy content are embedded in free text, as seen in the patterns discussed in the State of Secrets in AppSec and the Ultimate Guide to NHIs — Key Research and Survey Results. A practical AI programme also needs governance hooks, which is why standards such as ISO/IEC 42001:2023 AI Management System Standard are increasingly referenced for accountable management systems. In practice, many security teams encounter the real cost of unstructured data only after the first retrieval prototype returns inconsistent answers or exposes sensitive content that was never meant to be model input.

How It Works in Practice

AI programmes slow down when unstructured content has no agreed lifecycle: where it lives, who owns it, how it is classified, and what can safely be used for training, retrieval, or agent tools. The issue is not just format conversion. It is operational trust. Teams must resolve duplicates, remove stale content, redact secrets and personal data, and establish lineage so downstream users know whether a document, log file, or chat transcript is current enough to inform a model. Common control steps include:

• Inventorying unstructured sources by business domain, sensitivity, and retention rule.
• Applying classification and access policy before ingestion into RAG or analytics pipelines.
• Normalising content into searchable chunks while preserving source links and timestamps.
• Scanning for secrets, credentials, and regulated data before indexing or fine-tuning.
• Logging provenance so model outputs can be traced back to source material.

This is where AI security and identity governance intersect. If an AI agent can access documents, ticketing systems, or code repositories, its permissions need the same scrutiny as any other privileged workflow. That means binding access to least privilege, short-lived credentials, and auditable approvals, rather than letting an autonomous system inherit broad file access by default. Guidance from the State of Secrets in AppSec reinforces the point that hidden secrets and fragmented controls create avoidable clean-up work before AI can be trusted with real tasks. The practical goal is not perfect data cleanliness, but enough structure that the model can be governed and the output can be explained. These controls tend to break down when legacy repositories mix public, confidential, and operational content in the same folder structure because classification and ownership are too ambiguous to automate safely.

Common Variations and Edge Cases

Tighter data controls often increase preparation time and operating overhead, so organisations must balance speed of experimentation against the risk of training or querying on poor-quality content. Current guidance suggests that the right balance depends on use case criticality: a customer-support copilot may tolerate broader indexing than a regulated decision-support system, but there is no universal standard for this yet. In high-risk environments, content must be curated more aggressively, especially where DeepSeek breach-style failures show how exposed data and embedded secrets can scale the damage quickly. Edge cases often appear in:

• Multilingual content, where translation changes meaning and classification confidence.
• Scanned PDFs and images, where OCR errors distort context and produce false matches.
• Chat exports and collaboration tools, where informal language hides policy, legal, or security obligations.
• Agentic workflows, where an AI system can combine fragments across systems and surface information no single document intended to reveal.

For governance-heavy programmes, ISO/IEC 42001:2023 AI Management System Standard is useful as a management baseline, but it does not remove the need for source-level controls. The most common failure mode is assuming that data pipelines will fix ambiguity automatically. They do not when document ownership is unclear, retention is inconsistent, or security teams have not defined what “trusted source” means for the AI use case.

Related resources from NHI Mgmt Group

• Why do data quality problems become security problems in AI programmes
• How should security teams govern AI classification for unstructured data?
• How do security teams align AI governance with existing IAM and data security programmes?
• Why do AI security programmes need to connect access, data, and behaviour?