What do privacy teams get wrong about AI disclosures in privacy law?

Why This Matters for Security Teams

AI disclosure failures are rarely just a wording problem. Privacy teams can write a notice that mentions “AI” and still miss the operational obligations that matter: lawful basis, purpose limitation, data minimisation, retention, and a defensible process for rights requests. Under the EU General Data Protection Regulation (GDPR), disclosures are part of accountability, not a substitute for it.

The practical risk is that product, legal, security, and data teams each assume another group owns the assessment. That gap becomes more serious when models are trained on personal data, fine-tuned on internal records, or used to generate profiling outputs that affect individuals. NHI Management Group has also highlighted how fast exposed AI-related credentials can be abused in the wild in its LLMjacking research, which is a reminder that AI governance and access control cannot be separated from privacy governance.

In practice, many security teams encounter disclosure failures only after a DSAR, regulator inquiry, or model incident has already exposed the gap, rather than through intentional privacy-by-design review.

How It Works in Practice

Good AI disclosure practice starts with an inventory of where personal data enters the AI lifecycle and what the system does with it. That includes training, fine-tuning, retrieval, inference logs, human review, and downstream sharing. The disclosure should reflect the actual processing, but it should also be backed by documented assessments and technical controls. The NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it ties privacy intent to governance, auditability, and data handling controls.

For most organisations, the operational pattern is:

• Map each AI use case to the personal data categories involved.
• Record the role of the system: decision support, profiling, ranking, content generation, or automation.
• Document whether data is used for training, evaluation, monitoring, or only transient inference.
• Define who approves notice changes, DSAR handling, and retention exceptions.
• Ensure model outputs are reviewed when they can affect an individual’s rights or opportunities.

This matters because a notice can be technically accurate and still misleading if it omits material processing, such as reuse of prompts for training or logging for safety review. Current guidance suggests privacy teams should align disclosures with actual data flows, then test whether those disclosures remain correct after model updates, vendor changes, or new integration paths. NHIMG research on the DeepSeek breach underscores the downside of poor data governance when model ecosystems leak far more than intended. These controls tend to break down when AI is embedded in SaaS workflows with weak inventory ownership and logs that mix personal data, prompts, and secrets in the same telemetry stream.

Common Variations and Edge Cases

Tighter AI disclosure requirements often increase operational overhead, requiring organisations to balance transparency against change-management burden and legal review latency.

There is no universal standard for AI disclosures yet, so the right answer depends on jurisdiction, use case, and whether the system makes or materially influences decisions. Some organisations over-disclose by listing every conceivable AI interaction, which can dilute meaningful notice and create maintenance debt. Others under-disclose by treating vendor terms as sufficient, even when the organisation is the controller of the personal data processing.

Edge cases matter. If the model is used only for internal summarisation and never ingests personal data, the disclosure burden may be narrower. If the same system supports hiring, credit, pricing, fraud review, or health-related processing, the disclosure and assessment bar rises sharply. The most difficult cases are hybrid deployments where prompts, retrieval content, and user metadata all blur together. In those environments, privacy teams should treat notices as living artefacts linked to governance records, not static website copy. That approach is more resilient when model vendors, data processors, and internal owners change faster than policy review cycles.

Related resources from NHI Mgmt Group

• What do teams get wrong about AI security and access management?
• What do security teams get wrong about Shadow AI?
• What do security teams get wrong about automation bias in AI governance?
• What do security teams get wrong about passwordless authentication and AI risk?