AI security testing must verify runtime behaviour, not just model accuracy

At a glance

What this is: This is an analytical guide to AI security testing, with the central finding that runtime behaviour, not model accuracy, is where most enterprise AI risk emerges.

Why it matters: It matters to IAM and security practitioners because AI systems increasingly act under delegated identities, call APIs, and trigger workflows that need enforceable scope, attribution, and monitoring.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read LEVO's full analysis of AI security testing and runtime behaviour

Context

AI security testing exists because AI systems no longer behave like static software. They retrieve data, invoke APIs, and trigger workflows at runtime, so the security question shifts from whether a model is accurate to whether the system stays within approved boundaries when it is live. In AI security testing, the primary control problem is execution governance, not model quality.

That distinction matters for identity and access management because AI systems often operate through delegated credentials, service accounts, or shared identities. Once those identities can chain actions across data, tools, and downstream systems, traditional test methods are too narrow to show where policy, privilege, and runtime behaviour diverge. The article’s starting position is typical of modern enterprise AI deployments, not an edge case.

Key questions

Q: How should security teams test AI-powered systems in production-like environments before rollout?

A: Security teams should test AI systems across prompts, workflows, integrations, and safety boundaries before production exposure. Use realistic datasets, stress and latency checks, regression tests, and human review to catch nondeterministic failures. The goal is not exact output matching. It is to verify reliability, safety, cost, and behavior under conditions that resemble real users and real operational load.

Q: Why do delegated AI identities create governance risk?

A: Delegated AI identities can be broader than the task requires, and autonomous systems can chain valid actions into outcomes that violate policy or intent. That is why teams need least privilege, clear ownership, and revocation controls for service accounts, tokens, and shared credentials used by AI.

Q: What are the signs that an AI application is failing its security boundaries?

A: Common warning signs include inconsistent outputs for the same prompt, unexpected changes in response quality, hidden dependence on mutable retrieved data, and successful prompt manipulation through user controlled fields. In agentic systems, additional signs are unauthorized tool use, data exposure beyond intended scope, and actions that drift outside the original task boundary.

Q: How do organisations separate AI governance from AI security testing?

A: AI governance defines what should be allowed, while AI security testing verifies whether the deployed system actually stays within those boundaries. Governance without runtime validation is only policy on paper, especially once agents can retrieve data, call tools, and trigger workflows on their own.

Technical breakdown

Runtime AI security testing versus model evaluation

Runtime AI security testing examines what an AI system actually does in production, while model evaluation focuses on accuracy, bias, or robustness in controlled settings. The difference matters because a model can be technically sound and still create risk once it is connected to APIs, tools, data stores, and business workflows. Security testing therefore has to observe action scope, execution context, and policy enforcement at the point of use. That makes it closer to operational control validation than traditional application testing.

Practical implication: test live behaviour under real credentials and integrations, not just offline prompts or benchmark outputs.

Why delegated identities change the AI risk model

AI systems frequently act under service accounts, API keys, or shared tokens that let them perform tasks on behalf of the organisation. Those identities can be broader than the task requires, and once an AI system can chain multiple calls, it may stay within technical authentication while still violating intent. This is where NHI governance becomes relevant: machine identities need lifecycle, scope, and attribution controls just as much as human identities do, especially when actions are autonomous or semi-autonomous.

Practical implication: scope AI identities to task-level permissions and review them as machine identities, not as generic application access.

Behavioural drift and chained actions in AI systems

AI systems are not static after deployment. Prompts change, tools are added, APIs evolve, and context shifts, which means a system that was safe at launch can begin behaving outside policy without any obvious code change. Chained actions make this worse because each step can look legitimate in isolation while the sequence produces unsafe outcomes. That is why continuous testing matters. The control objective is not only to detect a single bad request, but to spot patterns of scope expansion before they become routine.

Practical implication: monitor for multi-step action paths, new tool combinations, and changes in access patterns over time.

Threat narrative

Attacker objective: The objective is to exploit legitimate AI execution paths so that harmful actions occur without obvious perimeter compromise or authentication failure.

1. Entry occurs when an AI system is connected to enterprise data and tools through delegated credentials that allow runtime actions.
2. Escalation happens when the system invokes APIs or chains actions beyond the original task scope while still appearing authenticated and authorised.
3. Impact follows when legitimate-looking execution leads to excessive data exposure, unsafe workflow completion, or policy violations at machine speed.

NHI Mgmt Group analysis

Runtime behaviour is now the security boundary for AI systems. The article correctly treats AI risk as an execution problem because deployed systems act, call, and chain decisions in ways static reviews cannot fully predict. That aligns with modern AI governance thinking, where policy intent only matters if the runtime system can be checked against it. Practitioners should treat observed behaviour as the control surface, not the model alone.

AI security testing and NHI governance are converging. When AI systems operate through service accounts, API keys, and delegated tokens, they become machine identities with real privilege. That means the same lifecycle questions used in NHI governance apply here: who owns the identity, what can it access, how is it revoked, and how is abuse detected. The field needs to stop treating AI execution identities as an edge case. Practitioners should govern them as first-class non-human identities.

Behavioural drift is the named concept practitioners should watch. AI systems can change security posture without a code release because prompts, tools, and integrations change over time. That creates governance debt: controls may appear intact while actual runtime behaviour moves outside approved boundaries. The implication for AI RMF and OWASP Agentic AI guidance is clear. Practitioners should continuously validate the system they have, not the system they initially approved.

Policy definitions alone do not secure autonomous execution. The article’s central warning is that a governance document cannot stop an AI system from taking unsafe actions if runtime enforcement is missing. This is especially true in environments where data access, tool invocation, and downstream workflows are loosely coupled. The discipline now needs measurable execution boundaries. Practitioners should prove policy at runtime, not assume it from design.

The identity of the AI system is becoming a control problem, not just an architecture detail. AI systems that can act without continuous human oversight require accountable ownership, scoped credentials, and revocation pathways. That makes identity governance part of AI security testing, not a separate programme. The practical conclusion is that security teams should align AI testing, NHI controls, and audit evidence into one governance model.

What this signals

AI security programmes are moving toward continuous execution assurance, not one-time assessment. That means teams need observable telemetry for tool calls, data access, and delegated identity use, plus a governance model that treats AI systems as operational actors with measurable boundaries.

Behavioural drift: once prompts, tools, or integrations change, the security posture of an AI system can change with them. Teams should pair runtime monitoring with policy validation, and align that work to the NIST AI Risk Management Framework so governance and enforcement stay connected.

For practitioners

• Define runtime test cases for AI execution scope Build tests that validate which APIs, tools, and data sources an AI system can actually reach under production credentials, then rerun them whenever integrations change.
• Treat AI service accounts as governed non-human identities Assign ownership, expiry, review cadence, and revocation criteria to every AI identity, including shared tokens and delegated service accounts used by agents.
• Monitor chained actions and policy drift continuously Track multi-step behaviour across sessions so teams can detect when an AI system begins combining valid actions into unsafe workflows or expanding its access pattern.
• Separate model assurance from execution assurance Keep model evaluation, red teaming, and governance documentation, but add runtime validation to confirm the AI system stays within approved operational boundaries.

Key takeaways

• AI security testing is really runtime execution testing, because the main risk appears after authentication, when systems begin calling tools and moving data.
• Delegated AI identities turn AI systems into non-human identity governance problems, which means scope, ownership, and revocation are now core controls.
• Continuous observation is the only reliable way to catch behavioural drift, chained actions, and policy violations before they become normal operations.

Key terms

• AI-driven security testing: AI-driven security testing uses automated analysis to discover vulnerabilities earlier in the software lifecycle. The practical value is not just speed, but earlier feedback while code, ownership, and context are still visible. It becomes useful when teams can repeat it continuously without creating extra operational friction.
• Runtime Visibility: The ability to observe what an AI client actually accessed, which tools it used, and how it behaved during a session. It is more useful than entitlement snapshots for agent governance because it captures executed reality, not just approved access.
• Behavioural Drift: Behavioural drift is the gradual change in what an identity does compared with what it was originally approved to do. For AI agents, drift can come from prompt changes, model updates, expanded integrations, or altered workflows, which makes access review alone an incomplete control.
• Delegated AI identity: Delegated AI identity describes an AI system that acts with permissions, credentials, or scoped authority on behalf of a business process. It matters because once an AI can initiate actions, it needs lifecycle, access, and offboarding controls similar to other governed non-human identities.

What's in the full article

LEVO's full research post covers the operational detail this post intentionally leaves for the source:

• Concrete runtime test scenarios for API invocation scope, data access, and action authorization
• The article's full breakdown of behavioural drift, chained actions, and failure handling in AI systems
• LEVO's explanation of how it distinguishes AI security testing from model risk management and governance
• Operational examples showing how runtime visibility supports continuous validation in deployed environments

👉 LEVO's full post covers runtime visibility, behavioural drift, and the control tests used to validate deployed AI systems.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to real-world operational risk.