AI black box risk: what it means for data and IAM teams

TL;DR: AI’s February 2025 post frames AI as a data and governance problem, not just a model problem, according to Cyera. The hard part is not visibility alone but controlling sensitive data access patterns as AI adoption expands in an environment where security teams must understand what the system can see, move, and expose.

NHIMG editorial — based on content published by Cyera: Opening the AI Black Box: Best Practices for Using AI in Cybersecurity

Questions worth separating out

Q: How should security teams govern AI systems that can access sensitive data?

A: Security teams should govern AI systems as access pathways, not just as software features.

Q: Why do AI systems create new identity and access risks?

A: AI systems create new identity and access risks because they often sit on top of powerful delegated credentials and data connectors.

Q: What breaks when AI security is treated separately from IAM?

A: What breaks is the control chain.

Practitioner guidance

• Inventory AI-connected identities and tokens Identify every service account, API key, token, and delegated access path used by AI workflows, then map each one to the data it can reach and the actions it can perform.
• Classify sensitive data before AI integration Mark the repositories, document stores, and knowledge bases that AI can query, then block high-risk classes from retrieval or summarization unless there is a documented business need.
• Tighten connector and retrieval permissions Review the permissions behind every connector, plugin, and retrieval layer so AI systems cannot expand into adjacent datasets through inherited access.

What's in the full article

Cyera's full blog post covers the operational detail this post intentionally leaves for the source:

• Data security posture management workflow examples for AI-connected repositories
• Platform framing for classifying and tracking sensitive data exposed to AI systems
• Operational discussion of access visibility and remediation around AI data paths
• Source-specific context on Cyera's AI security posture and platform modules

👉 Read Cyera’s analysis of opening the AI black box for cybersecurity teams →

AI black box risk: what it means for data and IAM teams?

Explore further

View Full Forum →  |  NHI Foundation Course →