AI disclosure and governance expectations are rising fast

At a glance

What this is: Collibra’s survey shows strong demand for AI disclosure, human oversight, and formal governance as organisations expand AI use.

Why it matters: For IAM and governance teams, the signal is that AI access, disclosure, and assurance now need to be treated as operating requirements across human, NHI, and autonomous programmes.

By the numbers:

• 93% agree that companies should be required to disclose their use of AI tools and agents to encourage transparency, monitoring and responsible use.
• 91% of decision-makers believe that human oversight for AI systems is critical.
• 64% consider it a "red flag" from a hiring perspective if a prospective employee cannot demonstrate familiarity with AI tools in an interview.
• 90% of decision-makers support federal requirements for companies to disclose and document high-risk AI systems.

👉 Read Collibra's survey findings on AI disclosure, oversight, and governance

Context

AI governance is moving from experimentation toward formal disclosure, oversight, and verification. In this survey, technology decision makers are not asking whether AI should be used, but whether its use should be visible, accountable, and tied to formal controls across the business.

That matters to identity teams because AI tools and agents now sit inside approval chains, data access paths, and decision support workflows. When governance lags behind adoption, organisations end up with opaque AI use, weak accountability, and a growing trust gap between what systems produce and what teams can confidently verify.

Key questions

Q: How should organisations disclose the use of AI tools and agents?

A: Organisations should maintain an inventory of approved AI tools and agents, record who owns them, and document what data and decisions they influence. Disclosure only works when it is tied to ownership, scope, and evidence. That gives security, privacy, and audit teams a reliable way to distinguish sanctioned use from shadow AI.

Q: Why does human oversight matter for AI governance?

A: Human oversight matters because AI outputs can look confident while still being wrong, biased, or incomplete. Oversight creates a decision boundary so the organisation knows when a person must review, approve, or override output before it affects access, compliance, or operational decisions. Without that boundary, accountability becomes unclear.

Q: How can security teams tell whether AI governance is working?

A: Teams can measure whether AI systems are documented, whether high-risk use cases have named owners, and whether review steps are actually followed before decisions are executed. If people still need to correct AI outputs manually without a formal process, governance is inconsistent and too dependent on individual judgement.

Q: Who is accountable when AI use is not disclosed?

A: Accountability should sit with the business owner, the system owner, and the governance function that approved or failed to inventory the AI use. If an organisation cannot name those parties, it does not have a controllable governance model. It has an exposure problem that will eventually surface in audit or incident response.

Technical breakdown

AI disclosure as a governance control

Disclosure means organisations can identify where AI tools and agents are being used, who approved them, and what data or decisions they influence. In identity terms, disclosure is not a communications exercise. It is a control that supports traceability, auditability, and accountability across access paths that may otherwise be invisible to IAM, IGA, or security teams. Without disclosure, governance cannot reliably distinguish sanctioned AI use from shadow AI activity.

Practical implication: establish an inventory of AI tools and agents with ownership, access scope, and business purpose.

Human oversight and formal verification

Human oversight is the check that prevents AI outputs from becoming de facto decisions without review. Formal verification of underlying data is the related governance step that makes those outputs trustworthy enough to act on. Together, they separate automation from delegated authority. In practice, this means oversight must be designed into workflows, not added as an afterthought once AI use becomes widespread.

Practical implication: map where human review is mandatory before AI output can trigger access, financial, or operational decisions.

AI literacy as part of identity governance

The survey’s hiring signal shows that AI literacy is becoming a governance requirement, not just a technical preference. Teams need people who can recognise AI risk, question output quality, and understand where AI intersects with data handling, access, and compliance. That is especially relevant where AI systems influence identity decisions, because weak literacy increases the chance that governance becomes ceremonial rather than effective.

Practical implication: include AI literacy and governance fluency in role expectations for IAM, privacy, and security decision makers.

Breaches seen in the wild

• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.
• Schneider Electric credentials breach — exposed credentials gave attackers access to Schneider Electric Jira, exfiltrating 40GB.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Disclosure is becoming the minimum viable control for AI governance. When 93% of decision makers want companies to disclose AI tools and agents, the market is signalling that hidden AI use is no longer acceptable in regulated or high-trust environments. Disclosure creates the starting point for monitoring, documentation, and accountability, which are prerequisites for any serious governance programme. Practitioners should treat undisclosed AI use as an identity visibility problem, not a communications issue.

The trust gap is really a verification gap. If 55% of decision makers sometimes have to personally push back or correct AI output, the issue is not just model quality. It is that organisations are delegating judgement into systems without sufficient data assurance, policy checks, or escalation paths. This is a governance failure across human and machine decision chains, and it shows why AI outputs cannot be treated as inherently authoritative.

AI literacy is now part of governance maturity. The finding that 64% view lack of AI familiarity as a hiring red flag shows that organisations now expect governance staff to understand AI behaviour well enough to challenge it. That expectation matters because data, identity, and compliance controls fail when the people operating them cannot recognise AI-driven risk. Practitioners should align role design, training, and access review responsibilities with AI competence.

Formal governance is the bridge between adoption and accountability. With 90% supporting disclosure and documentation for high-risk AI systems, the sector is moving toward evidence-based governance rather than informal reliance on policy statements. This does not replace identity controls, but it does reframe them: who can use AI, what they can connect it to, and what is recorded when they do. Practitioners should expect auditability to become a default requirement, not a special case.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• That behaviour gap matters here because AI governance breaks in the same way as secrets governance, when confidence outpaces verification and control evidence cannot keep up with usage.
• Lifecycle Processes for Managing NHIs is the right next step when teams need to connect disclosure, ownership, rotation, and offboarding into one governance model.

What this signals

Disclosure will become a control expectation, not a policy preference. As AI use moves deeper into business workflows, organisations will need to prove where AI is used, who approved it, and what controls surround it. That is especially true where AI systems intersect with identity, access, and regulated decisions. Teams should prepare for governance questions that are evidence-led rather than statement-led.

AI literacy is starting to shape role design across governance functions. The 64% hiring signal suggests that AI fluency will increasingly be treated as part of competence for IAM, privacy, and security decision makers. For practitioners, that means recertification, access review, and policy ownership will need to account for whether reviewers can actually evaluate AI behaviour. The right benchmark is not enthusiasm for AI, but governance competence.

With 27 days to remediate a leaked secret in our research, the governance lesson is clear: confidence without verification creates blind spots that scale faster than remediation can respond. See The State of Secrets in AppSec for the underlying pattern, then map the same discipline to AI tool disclosure and oversight.

For practitioners

• Inventory AI tools and agents by owner and access path Build a living register of sanctioned and unsanctioned AI use, including the data sources, accounts, and workflows each system can touch. Tie each entry to a named business owner and review cadence.
• Define mandatory human review points for high-impact AI output Require review before AI-generated output can affect access decisions, compliance determinations, or customer-facing actions. Use approval gates for the points where the organisation cannot tolerate silent model error.
• Add AI literacy to governance roles and recertification criteria Update IAM, privacy, and security role profiles so reviewers can identify AI risk, challenge outputs, and understand where AI changes access and accountability assumptions.
• Document high-risk AI systems for audit and oversight Record what the system does, what data it consumes, who can change it, and which controls evidence its use. This makes disclosure operational instead of symbolic and gives auditors a traceable control set to test.

Key takeaways

• AI governance is shifting from optional experimentation to mandatory disclosure, oversight, and evidence.
• The strongest signal in the survey is not adoption optimism but the trust gap between AI output and verifiable control.
• IAM and governance teams should treat AI literacy, disclosure, and human review as part of the operating model, not a side programme.

Key terms

• AI Disclosure: AI disclosure is the practice of documenting where AI tools and agents are used, what they can access, and who is accountable for their operation. It turns hidden or informal AI use into something governance, audit, and security teams can verify and review.
• Human Oversight: Human oversight is the requirement that a person remains responsible for reviewing, approving, or correcting AI-driven output before it causes a material action. In governance terms, it is the control that prevents automation from becoming unowned authority.
• Shadow AI: Shadow AI is AI use that exists outside approved governance, usually because the organisation has not inventoried it, assigned ownership, or defined review boundaries. It often appears first in employee workflows and becomes a visibility problem before it becomes a policy problem.
• Formal Governance Framework: A formal governance framework is the documented set of ownership, review, escalation, and evidence requirements that makes AI use auditable. It matters because confidence in AI systems is not a control unless the organisation can prove how decisions were made and by whom.

Deepen your knowledge

AI disclosure, oversight, and governance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If your team is trying to bring AI use under a controllable identity and governance model, it is worth exploring.

This post draws on content published by Collibra: More Than 90% Agree that Companies Should be Required to Disclose Use of AI Agents. Read the original.