AI trust is weakening identity controls and social engineering

At a glance

What this is: This is an analysis of how AI is amplifying social engineering, shadow AI, and identity risk in enterprise environments.

Why it matters: It matters because IAM and NHI controls have to verify both human and machine identities when AI output, access, and decision-making are increasingly trusted by default.

👉 Read CyberArk's analysis of AI trust, social engineering, and shadow AI

Context

AI use is changing the trust model in security operations and day-to-day work. The primary risk is not just faster content generation, but a quieter loss of critical checking that lets convincing output, credential misuse, and unsanctioned tools blend into normal workflows. For IAM and NHI governance, that means identity assurance has to keep pace with how people and systems actually make decisions.

Shadow AI is the clearest governance gap in this article. When employees route sensitive data into unapproved models or accept AI output without verification, the organisation loses control over both secrets and the identity context attached to that data. That starting position is now typical, not exceptional, in enterprises trying to adopt AI faster than their controls mature.

Key questions

Q: How should organisations govern shadow AI without blocking legitimate use?

A: Start with approved-use policy, tool inventory, and data classification. Then require that any AI system handling internal information has named owners, logged access, and defined credential paths. The goal is not prohibition, but visibility and control. If a tool cannot be inventoried or monitored, it should not process sensitive data.

Q: Why does AI make social engineering harder to spot?

A: AI lets attackers tailor messages with context, tone, and timing that resemble normal internal communication. That reduces the usefulness of generic phishing cues. Security teams should assume that convincing language is no longer evidence of legitimacy and pair user awareness with identity-based verification for risky requests.

Q: What is the difference between AI content risk and AI identity risk?

A: AI content risk concerns incorrect, biased, or harmful output. AI identity risk concerns who or what is allowed to generate, retrieve, or act on data in the first place. Identity risk is usually more operationally dangerous because it can lead directly to credential abuse, data exposure, and unauthorized action.

Q: When should teams use just-in-time access for AI-related systems?

A: Use just-in-time access whenever an AI-connected workflow can reach sensitive data, infrastructure, or privileged functions. The access should be time-bound, task-bound, and auditable. If the workflow does not need persistent privilege to operate, standing access creates unnecessary exposure and should be removed.

Technical breakdown

How AI changes social engineering mechanics

AI shifts phishing and impersonation from broad, low-effort campaigns to highly tailored attacks. Attackers can combine public data, role context, and writing style to make a message look like it came from a trusted internal source. The technical risk is not only better prose, but better targeting and timing. That reduces the value of user intuition alone and increases the need for layered verification at the identity and transaction level.

Practical implication: Treat convincing content as insufficient proof of legitimacy and require step-up verification for high-risk requests.

Why shadow AI becomes an identity governance problem

Shadow AI is a governance issue because it creates unsanctioned pathways for data access, processing, and retention. Once employees feed internal information into external models, the organisation may lose visibility into where sensitive content and associated secrets go next. The NHI angle is direct: machine-to-machine flows, API keys, and model integrations can all bypass normal approval paths if they are not inventoried and controlled.

Practical implication: Build an approved AI usage inventory that covers tools, data classes, and the credentials used to connect them.

Why zero trust depends on human and machine verification

Zero trust only works if identity checks are continuous and context-aware. In AI-heavy environments, that means validating the human requester, the workload or agent acting on their behalf, and the sensitivity of the action being attempted. Standing access and default privilege create the conditions for abuse because they let malicious or mistaken actions happen without fresh authorization. Ephemeral access and tighter policy enforcement narrow that window.

Practical implication: Apply zero standing privilege to AI-connected workflows and require just-in-time access for elevated actions.

Breaches seen in the wild

• Cisco DevHub NHI breach — IntelBroker exploited exposed Cisco credentials, API tokens and keys in DevHub.
• MongoBleed breach — MongoBleed exposed secrets across 87K MongoDB servers.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI trust debt is now an identity security issue. The article shows that organisations are not only consuming more AI output, they are also gradually reducing the amount of human challenge applied to that output. That creates a form of trust debt where convenience accumulates faster than verification. For NHI governance, this matters because the same cultural drift that accepts AI output can also normalise machine access without proper review.

Shadow AI widens the non-human identity attack surface. Unsanctioned AI tools create hidden data flows, hidden integrations, and hidden credentials. Those paths are difficult to govern with traditional app onboarding or periodic access review alone because they appear and disappear in response to user behaviour. The result is a governance gap between approved IAM policy and actual AI usage, and that gap should be treated as exposure until proven otherwise.

Identity verification must extend to AI-mediated actions. The article rightly returns to least privilege and zero trust, but AI changes the operational meaning of both. Teams need to ask not only who requested access, but what system is acting, what data is being processed, and whether the action is reversible. NHI controls are no longer just about authentication. They are about constraining autonomous or semi-autonomous execution before it becomes routine.

Ephemeral credential trust debt: organisations adopting AI without inventorying credentials, data paths, and approval workflows accumulate invisible exposure. The debt shows up later as unaudited access, unsanctioned tools, and weak accountability. Practitioners should treat AI governance as an identity programme, not a content-safety add-on.

From our research:

• 59.8% of organisations see value in a solution that simplifies non-human access management and introduces dynamic ephemeral credentials, according to The 2024 Non-Human Identity Security Report.
• Only 19.6% of security professionals express strong confidence in their organisation's ability to securely manage non-human workload identities, a useful baseline for the current maturity gap.
• For a deeper control lens, compare that maturity gap with The 52 NHI breaches Report to see how access failures become real incidents.

What this signals

Trust drift is the operational risk that security teams should watch. As AI becomes a default assistant, the organisation’s willingness to verify declines unless controls force a pause. That matters for NHI programmes because the same pattern appears when machine access is easy to grant and hard to review. Teams should expect more workflow-level abuse, not just more email phishing.

Only 19.6% of security professionals express strong confidence in their organisation's ability to securely manage non-human workload identities, according to The 2024 Non-Human Identity Security Report, which suggests the challenge is structural rather than tactical. If you are adding AI into daily operations, the identity programme needs to own the access path, the approval path, and the audit path together.

Security leaders should prepare for AI governance reviews to merge with IAM and secrets management reviews. That means treating AI tools as workloads, not just users, and applying the same scrutiny to tokens, service accounts, and delegated permissions that already exists for other NHI classes.

For practitioners

• Inventory AI tools and connected credentials Map every approved and unapproved AI tool, then record the data classes, API keys, service accounts, and tokens each one can reach. Include browser plug-ins and personal accounts used for work tasks. The goal is to make shadow AI visible before it becomes a durable control gap.
• Require human verification for high-risk AI-assisted actions Set a policy that AI-generated output cannot trigger privileged changes, payments, account resets, or security exceptions without independent human review. Use step-up approval when the output is used to justify access or operational decisions.
• Apply just-in-time access to AI-connected workflows Replace standing privileges with task-scoped access for systems that read, generate, or transform sensitive data. Tie access to a narrow time window and explicit business purpose so compromised identities have less opportunity to abuse AI-assisted automation.
• Monitor for anomalous AI usage and credential reuse Look for repeated prompts, unusual data exports, new model endpoints, and credentials used outside normal application paths. Correlate that telemetry with identity logs so you can distinguish routine automation from unsanctioned behaviour.

Key takeaways

• AI adoption is weakening verification habits, and that creates a direct identity security problem for both people and machines.
• Shadow AI expands the NHI attack surface by introducing hidden credentials, data flows, and approval paths that normal governance often misses.
• The practical response is to inventory AI connections, remove standing privilege, and require human verification before high-risk actions execute.

Key terms

• Shadow AI: Shadow AI is the use of unsanctioned AI tools or models inside an organisation, usually outside approved governance and monitoring. It matters because it creates hidden data flows, hidden credentials, and hidden decision paths that security teams cannot easily review or revoke.
• Just-in-time access: Just-in-time access is a privilege model where credentials or permissions are granted only when a specific task needs them and for a limited time. For NHI governance, it reduces the window in which stolen or misused credentials can be abused.
• Zero standing privilege: Zero standing privilege means no identity, human or non-human, keeps persistent elevated access by default. Access is provisioned on demand, tightly scoped, and removed when the task ends. This reduces blast radius and makes abuse harder to sustain.
• Identity verification: Identity verification is the process of confirming that a user, workload, or agent is the entity it claims to be before access is granted. In AI-heavy environments, that verification must include the requester, the system acting on its behalf, and the sensitivity of the action.

Deepen your knowledge

AI trust, shadow AI, and identity verification are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building governance around AI-connected workflows, it is a practical place to start.

This post draws on content published by CyberArk: Are we trusting AI too much? Read the original.