AI trust is breaking across agents, models, and content

At a glance

What this is: This is an analysis of how AI is degrading trust across agents, models, and synthetic content, with the core finding that existing governance assumptions no longer hold.

Why it matters: It matters because IAM, NHI, and security teams now have to govern AI actors, verify provenance, and control trust boundaries across both human and machine-driven workflows.

By the numbers:

• It's now present in more than 90% of organizations as employees increasingly rely on unapproved AI tools.
• Just 40% of individual consumers consider AI a trustworthy source of information.
• A staggering 70% don't trust companies to use AI responsibly with their data.

👉 Read DigiCert's analysis of trust degradation across AI agents, models, and content

Context

AI trust is the ability to verify what an AI system is doing, what data it can reach, and whether its outputs can be relied on. In this case, the primary problem is that AI is expanding the trust surface faster than governance, visibility, and accountability can keep up.

That matters to IAM and NHI programmes because the same identity assumptions that work for people and static workloads do not hold when software can act, decide, and generate content at scale. The article argues that organisations now need verifiable trust across agents, models, and content, not just stronger policy language.

The shift is already showing up in shadow AI, synthetic media, and concerns about accountability when systems make decisions without clear ownership. For many enterprises, that starting position is typical, not exceptional.

Key questions

Q: How should security teams govern shadow AI without blocking business use?

A: Start by identifying where shadow AI already exists, then classify the data, identities, and business processes it touches. Allow only approved tools to handle sensitive information, and require logging, review, and ownership for any AI that can affect decisions or move data outside managed boundaries.

Q: Why do AI agents create governance problems for IAM teams?

A: AI agents can act, select tasks, and process data with limited supervision, so standard access records do not always capture who initiated the action or why. IAM teams need identity, authorization, and auditability that attach directly to agent behaviour, not just to the human who deployed it.

Q: How can organisations prove content authenticity in an AI-heavy environment?

A: Use cryptographic provenance controls that preserve origin and modification history from creation through distribution. That matters most for media, customer communications, legal evidence, and any output where impersonation or alteration could create fraud, compliance, or reputational damage.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should sit with the organisation that approved the system, defined its operating boundaries, and accepted the risk. If no clear owner exists for approvals, monitoring, and remediation, the governance model is already too weak to support trustworthy AI.

Technical breakdown

Shadow AI and the trust surface problem

Shadow AI expands the trust surface because unapproved tools move data, prompts, and outputs outside sanctioned identity controls. The issue is not only policy violation, but loss of visibility into where information is processed and reused. In identity terms, the organisation can no longer reliably answer who or what initiated the action, which system handled the data, or how the resulting artefact should be governed. That makes traditional access control incomplete, because control only exists where the organisation can see and bind the actor to policy.

Practical implication: map unapproved AI use to data access paths, not just application risk, so governance can follow the identity trail.

AI agents need identity, authorization, and auditability

AI agents are different from ordinary automation because they execute tasks, access systems, and make decisions with limited supervision. That creates an identity problem, not just an operations problem. If the agent has no strong identity, the enterprise cannot attach authorization boundaries or audit responsibility to its actions. If the agent can act but cannot be traced cleanly, governance breaks at the point of accountability. This is why agent governance must be treated as a lifecycle and access problem across provisioning, authorization, and logging.

Practical implication: require cryptographic identity, scoped authorization, and durable logs for every agent that can touch business systems.

Content authenticity now depends on cryptographic provenance

Synthetic media undermines the old assumption that visual or written content is inherently trustworthy. Content authenticity now depends on provenance signals that survive distribution, editing, and reuse. Standards such as C2PA address that gap by attaching metadata that can be verified independently, giving consumers and enterprises a way to assess origin and modification history. For security teams, this is relevant because trust is no longer only about who accessed a system. It is also about whether the output a system produced can be proven authentic after the fact.

Practical implication: validate provenance controls for high-risk communications, especially where brand, legal, or fraud exposure depends on authenticity.

Threat narrative

Attacker objective: The objective is to exploit weak trust boundaries to gain unauthorised access to data, generate convincing synthetic output, and bypass accountability.

1. Entry occurs when employees adopt unapproved AI tools and move sensitive inputs into systems that are outside sanctioned governance and logging boundaries.
2. Escalation happens when agents or models reuse data, generate outputs, or make decisions without clear identity, accountability, or fine-grained authorization.
3. Impact follows as trust degrades across operations, compliance, and customer communications, while synthetic content and opaque automation increase fraud and reputational exposure.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Trust in AI has become an identity problem, not a perception problem. The article shows that the trust surface now spans agents, models, and content, which means governance cannot stop at policy statements or usage guidance. When an AI actor can initiate actions, process data, and generate outputs, the question becomes whether identity, authorization, and audit can still bind the behaviour to an accountable owner. Practitioners should treat trust as a control plane, not a brand promise.

Shadow AI is the clearest sign that governance is losing the race to adoption. Once more than 90% of organisations have unapproved AI use in play, the issue is no longer whether AI is present, but whether it is visible, traceable, and controllable. That is a classic NHI governance failure pattern, except the actor surface now includes tools that can reason over data and act on behalf of users. Practitioners should assume the gap is structural until identity controls reach the point of use.

Content authenticity is becoming part of identity governance. Synthetic media turns provenance into a security control because the organisation must prove origin and modification history after creation. That widens the remit of IAM and security teams, who now have to think about authenticity for communications, evidence, and customer-facing outputs. Practitioners should align verification controls with the risk of impersonation, fraud, and regulatory challenge.

AI agents force lifecycle governance to move from static access to governed runtime behaviour. Agents are not just another workload because they can execute, choose actions, and operate with limited supervision across business systems. The governance model has to account for their identity, their operational lifecycle, and their decision boundaries, or else accountability becomes too diffuse to enforce. Practitioners should reframe agents as governed entities, not just tools with a UI.

Trust degradation is now a cross-domain security signal. The same organisation can face shadow AI inside, deepfake-driven fraud outside, and model integrity concerns in between. That means AI trust cannot be isolated in a single team or control framework. Practitioners should coordinate IAM, NHI, fraud, compliance, and security architecture around one shared trust model.

From our research:

• The number of deepfake files has exploded from roughly 500,000 in 2023 to more than 8 million in 2025, according to The State of Secrets in AppSec.
• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases.
• If you are building governance for AI systems, see OWASP Agentic AI Top 10 for the control failures that emerge once tools, memory, and actions converge.

What this signals

Content authenticity is becoming a governance requirement, not a communications preference. As synthetic media becomes cheaper to generate and harder to detect, legal, fraud, and security teams will need common verification patterns for outputs that can affect customers or regulators. The practical shift is toward provenance, review, and policy-backed publication workflows rather than trust by default.

The scale of unapproved AI use means many programmes will discover AI risk through data incidents before they discover it through architecture reviews. That makes discovery and classification the first control step, especially where AI use overlaps with sensitive repositories, external sharing, or regulated records.

For teams building a broader identity programme, the lesson is that AI trust spans human, machine, and emerging agentic workflows in one operating model. That is why identity governance has to be coupled to provenance and accountability controls, not treated as a standalone access review exercise.

For practitioners

• Inventory unapproved AI usage paths Map where employees are already using shadow AI, then tie each tool to the data types, systems, and identities involved. Focus on where sensitive content leaves approved logging and access boundaries.
• Bind AI agents to cryptographic identity Require each agent that can act on business systems to have a clear identity, scoped authorization, and an audit trail that preserves responsibility across the full action path.
• Verify provenance for high-risk content Adopt authenticity checks for customer communications, evidence records, and brand-sensitive media so you can validate source and modification history before distribution.
• Rework accountability for AI-driven decisions Define who owns approvals, exceptions, and remediation when AI systems make or influence decisions, then document those responsibilities in governance and incident workflows.

Key takeaways

• AI trust is breaking because organisations can no longer assume they know which systems are acting, which data they are using, or who owns the outcome.
• Shadow AI, deepfake growth, and low consumer trust show that the problem is already at scale, not on the horizon.
• Security and IAM teams need identity binding, provenance verification, and accountable governance if they want AI adoption without uncontrolled trust loss.

Key terms

• Shadow AI: Shadow AI is the use of AI tools or services outside approved governance, visibility, or security controls. In practice, it creates unmanaged data movement, unclear ownership, and hidden trust exposure because the organisation cannot reliably trace what was sent, processed, or reused.
• AI Agent Identity: AI agent identity is the set of controls that ties an agent to a known, accountable runtime presence. It includes authentication, authorization, and logging so the organisation can distinguish one agent from another and enforce policy on what each agent may do.
• Content Authenticity: Content authenticity is the ability to verify that digital media or text came from a trusted source and has not been altered in unauthorised ways. It relies on provenance, signing, and verification mechanisms that preserve evidence of origin and change history.
• Trust Surface: The trust surface is the total area where an organisation must establish or maintain confidence in AI behaviour, outputs, and handling of data. It includes agents, models, content, and the systems that move information between them.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by DigiCert: The degradation of trust in the age of AI. Read the original.