Enterprise AI security needs continuous testing across the full stack

At a glance

What this is: This is an analysis of why enterprise AI applications create a broader attack surface than conventional software, with exploitability emerging across retrieval, memory, tools, permissions, and workflow execution.

Why it matters: It matters because IAM, PAM, and security architecture teams must govern what AI systems can access and do, not just whether the model answers correctly or appears safe in testing.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read terra's analysis of enterprise AI application security and exploitability

Context

Enterprise AI security fails when teams treat copilots, chatbots, and autonomous workflows like ordinary applications. These systems combine retrieval, memory, tool use, and delegated permissions, so the real governance problem is not just model safety but what the system can be manipulated to do.

That distinction matters for IAM and NHI programmes because AI systems increasingly behave like non-human identities with access to internal data and business actions. Once an application can read context, call APIs, and trigger downstream workflows, security teams need visibility into identity, privilege, and authorization across the full execution path.

Key questions

Q: What breaks when AI agents are treated like standard human users?

A: You lose visibility into effective permissions, expected behaviour, and real blast radius. Human-centric controls can misclassify normal agent activity as compromise, or miss policy violations that happen entirely within legitimate access. The failure is not only technical, it is governance design that assumes a person is always behind the action.

Q: Why do AI agents create a governance problem for IAM teams?

A: AI agents create a governance problem because they authenticate and act as autonomous software entities with tool access. If their actions are logged only as application activity, teams lose accountability, context, and revocation clarity. IAM must therefore extend to agent identity, delegated authority, and control-plane audit trails.

Q: How do security teams know whether AI exploitability is actually being reduced?

A: Look for evidence that manipulated prompts no longer translate into tool calls, data exposure, or workflow actions. A useful signal is whether the system enforces separate authorization for each sensitive step and whether those checks still hold after prompt, model, or integration changes. If controls only catch bad output, exploitability remains.

Q: What frameworks should organisations use to assess agentic AI risk?

A: Use OWASP Agentic AI Top 10 for threat modelling, OWASP NHI guidance for credential and privilege governance, and Zero Trust principles for continuous verification of tools and identities. Together, they help teams evaluate agent behaviour, access boundaries, and trust at runtime instead of only at onboarding.

Technical breakdown

Why AI application security breaks traditional AppSec boundaries

Conventional AppSec assumes a bounded request-response model, where the main question is whether input is handled safely and output is correct. AI applications break that assumption because the system can retrieve data, maintain memory, call tools, and act on instructions across multiple layers. The exploitable surface is therefore not just code, but the combination of prompt chains, context sources, permissions, and downstream business logic. That creates a gap between model evaluation and real-world exploitability, especially when an attacker can influence what the system sees and how it acts.

Practical implication: test the full execution path, not just the model, when validating AI application risk.

How tool use, memory, and permissions create AI exploitability

Agentic systems become risky when language understanding is connected to action. A prompt can alter the model’s interpretation, but the damage happens when that altered interpretation drives tool execution, data retrieval, or workflow changes under delegated permissions. Memory adds persistence to the problem because it can carry manipulated context forward across sessions or tasks. In practice, this means the control boundary is no longer a single model call. It is the chain from input to retrieval to authorization to action, and each link can be abused differently.

Practical implication: separate prompt handling from action authority and constrain each tool with explicit authorization checks.

Why continuous AI security testing is becoming necessary

Static scanning and one-off red teaming miss a core property of enterprise AI systems, which is that prompts, integrations, models, and workflows change constantly. A weakness that looks theoretical in a lab can become exploitable once it is connected to internal systems or production data. Continuous validation is therefore less about finding every possible issue and more about proving whether a weakness can trigger real business impact in the current environment. That is why exploitability testing has to cover input handling, retrieval, memory, permissions, and downstream outcomes together.

Practical implication: establish continuous validation across AI pipelines so changes in prompts, tools, or access do not create unseen exposure.

Threat narrative

Attacker objective: The attacker wants the AI system to carry out actions, disclose data, or cross authorization boundaries that users and defenders never intended.

1. Entry occurs when an attacker manipulates prompts, embedded content, or third-party context to influence an AI system’s interpretation.
2. Escalation follows when the altered interpretation reaches tool use, retrieval, memory, or delegated permissions that let the system act beyond intent.
3. Impact occurs when the AI system exposes restricted data, crosses a tenant boundary, or triggers an unsafe workflow action in production.

NHI Mgmt Group analysis

AI exploitability, not model correctness, is the security problem. The article is right to separate unsafe output from unsafe outcome. Enterprise AI risk emerges when manipulated context is converted into data access, tool execution, or workflow action. That makes the issue an authorization and governance problem as much as a model risk problem. Practitioners should treat exploitability as the primary control objective.

AI systems are becoming non-human identities in practice, even when teams do not label them that way. Once a copilot or agent can access internal systems and act with delegated permissions, it behaves like an identity-bearing workload. That creates a direct bridge between AI governance and IAM, PAM, and NHI controls. The right question is not whether the model is accurate, but whether the system’s identity, privileges, and action scope are bounded.

Continuous validation is the only realistic control model for dynamic AI environments. AI systems mutate through prompt changes, retrieval updates, new tools, and workflow revisions, so one-time assessments age quickly. This is where AI security governance starts to resemble runtime access governance. Practitioners should assume that today’s safe path may become tomorrow’s exploit path after a minor integration change.

AI governance debt is building faster than most security programmes can absorb it. Organisations are deploying AI into business processes before they have visibility into what the systems can access, what they can trigger, or who owns the resulting risk. That gap will keep widening unless security teams align AI oversight with identity, authorization, and change management disciplines. Practitioners should treat governance before scale as the default stance.

OWASP Agentic Applications Top 10 is becoming a practical reference point, not a theoretical one. The article’s discussion of tool misuse, privilege abuse, and memory poisoning aligns with the emerging agentic AI threat model. Those categories matter because they describe how AI systems fail in production, not just how models misbehave in tests. Practitioners should use that vocabulary to structure controls and red-team objectives.

What this signals

AI governance debt will surface first as identity and authorization debt. As copilots and agents move deeper into enterprise processes, teams will discover that the weakest point is not model quality but access scope. Security leaders should expect AI programmes to drive new review work across service accounts, delegated permissions, and tool authorization, especially where AI systems can cross system boundaries.

The practical next step is to treat AI changes like privilege changes. When prompts, retrieval sources, workflows, or agent permissions shift, the organisation should re-validate not only model behaviour but also whether the AI system still sits inside its intended access boundary. That is where identity governance and AI security converge.

Enterprise teams should also align AI security with the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 so the conversation moves from abstract model safety to operational controls. The organisations that do this well will be able to prove what AI can access, what it can trigger, and who owns the resulting risk.

For practitioners

• Map AI execution paths end to end Document how prompts, retrieval layers, memory stores, tools, and downstream workflows connect, then identify where a manipulated input can become an unauthorised action. This mapping should include data sources, API calls, and business process triggers so teams can see where exploitability begins.
• Separate model access from action authority Treat model output as advisory unless a separately enforced control authorises the next step. Require explicit checks before tool use, data retrieval, or workflow execution, especially where AI systems can act under delegated permissions.
• Apply continuous validation to AI changes Re-test AI systems whenever prompts, integrations, models, or permissions change, because each update can create a new exploit path. Use adversarial testing to verify whether manipulated context can trigger real business impact.
• Bound delegated permissions for copilots and agents Reduce standing access for AI systems and give them only the narrowest permissions needed for the task. Review whether internal copilots inherit excessive access from service accounts, workflow engines, or application roles.
• Link AI security reviews to IAM and NHI governance Classify AI systems that call tools or access internal data as identity-bearing systems, then bring them into access review, secrets handling, and privilege governance. Use the NHI lens where an AI system functions like a workload identity.

Key takeaways

• AI systems expand the attack surface when they can retrieve data, call tools, and trigger workflows under delegated permissions.
• The most consequential risk is exploitability across the full execution chain, not merely unsafe model output.
• Security teams need continuous validation, identity-aware controls, and separate authorization for AI actions to keep pace with change.

Key terms

• Application exploitability: Application exploitability is the degree to which an observed defect can be turned into a working attack against a live system. It depends on deployment context, authentication paths, configuration, and runtime behaviour. A finding becomes operationally important when an attacker can reach it and use it for access, collection, or exfiltration.
• Delegated permissions: Delegated permissions are access rights granted through another identity, application, or administrative relationship rather than directly to the end user. They matter because revoking the visible account does not always remove the underlying authority, which can preserve access after offboarding.
• Identity-bearing operational system: A system that can authenticate, query other services, or trigger operational actions in ways that require lifecycle ownership and access governance. In security operations, that includes agents, service accounts, and integrations that influence case handling or response.
• Continuous AI validation: An ongoing testing approach that rechecks AI behaviour and exploitability whenever prompts, tools, models, or permissions change. It is stronger than one-off assessment because enterprise AI systems evolve quickly and can become vulnerable after routine updates.

What's in the full article

terra's full article covers the operational detail this post intentionally leaves for the source:

• Terra’s testing approach for validating exploitability across input, retrieval, memory, tools, and downstream workflows.
• Examples of AI vulnerabilities observed in production environments, including prompt injection, tenant boundary issues, and workflow abuse.
• How the vendor frames continuous penetration testing for AI-enabled systems versus one-off model assessment.
• The article’s discussion of what security leaders should ask before AI is allowed to act autonomously.

👉 The full terra article covers continuous AI testing, exploitability validation, and the limits of traditional AppSec for AI systems.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to modern automation and AI-driven access patterns.