AI security testing must verify runtime behaviour, not just model accuracy

At a glance

What this is: This analysis says AI security testing is about runtime behaviour, not just model accuracy, because deployed AI systems can call APIs, access data, and execute actions autonomously.

Why it matters: It matters because identity, authorization, and monitoring controls must prove AI systems stay inside approved boundaries after deployment, not only in pre-release evaluation.

Context

AI security testing is a control problem, not just a model-quality problem. Once AI systems can retrieve records, invoke APIs, and trigger workflows, the real risk shifts to whether those actions stay inside approved business and security boundaries.

That shift matters for IAM, PAM, and NHI governance because an AI system often acts through delegated identities, service accounts, tokens, or other non-human credentials. If runtime behaviour is not checked continuously, a technically valid action can still become an unauthorized one.

The article frames that gap correctly: traditional testing can prove a system is functional, but it cannot by itself prove that autonomous behaviour remains safe after deployment. That makes runtime validation a governance requirement rather than a nice-to-have maturity step.

Key questions

Q: What breaks when AI monitoring only looks at model accuracy?

A: Accuracy alone can hide serious problems such as unfair outcomes, cohort-specific failures, and behaviour changes after deployment. A model can remain statistically strong overall while performing poorly for particular groups or in specific contexts. Effective governance therefore needs explainability, bias analysis, and operational monitoring together, so teams can see whether the model is reliable and acceptable in practice.

Q: Why do delegated credentials increase risk when AI agents and users are not clearly separated?

A: Delegated credentials can blur the boundary between user intent and agent autonomy. If the agent inherits the user’s identity, it may execute actions that are technically authorized but operationally unsafe, such as deleting production resources or exposing sensitive data. Clear identity separation allows additional policy checks, runtime controls, and confirmation steps before high-risk actions proceed.

Q: What are the signs that an AI agent session is drifting out of policy?

A: Look for unexpected tool calls, suspicious command sequences, credential exposure attempts, memory manipulation, or changes that exceed the user's stated objective. Those signals indicate that the agent is no longer acting within the intended task boundary. The key is to evaluate the whole sequence, not any single isolated event.

Q: How should teams govern AI agents that can reach APIs, events, and memory?

A: Teams should govern those agents as runtime identities, not as isolated integrations. That means enforcing policy at execution time, logging every tool and data access, and binding actions back to a clear initiating workflow or identity. If the control plane cannot show who acted, what they reached, and why, the programme does not have usable governance.

Technical breakdown

Why runtime AI security testing is different from model evaluation

Model evaluation asks whether outputs are accurate, robust, or aligned in a controlled setting. Runtime AI security testing asks a different question: what does the system actually do when it has live credentials, real data, and access to tools? That distinction matters because AI systems can be technically correct and still operationally unsafe if they cross authorization boundaries, call privileged APIs, or combine valid actions into an unsafe sequence. Security teams need to test execution context, not only prediction quality.

Practical implication: Test live API calls, data access, and action scope in the environments where the AI will actually operate.

How delegated identities turn AI behaviour into an IAM problem

AI agents rarely act as anonymous software. They usually execute through delegated credentials, service accounts, or shared identities that determine what they can see and do. Once those identities are in play, the security question becomes one of entitlement control, auditability, and least privilege. If the identity can reach too many APIs or broader datasets than the task requires, runtime behaviour can appear legitimate while still violating policy. This is where AI security testing intersects directly with IAM and NHI governance.

Practical implication: Verify that every AI action is bound to a scoped identity and that the identity cannot exceed the approved task boundary.

Why behavioural drift breaks one-time security sign-off

AI systems are not static after deployment. Prompts change, tools are added, permissions drift, and upstream data sources evolve. A one-time assessment therefore gives only a snapshot of a moving target. Behavioural drift is especially dangerous because unsafe access patterns can emerge gradually, without a visible release event or a clear configuration change. Runtime testing has to be continuous enough to catch those shifts, otherwise organisations will assume a control still works after its operating context has changed.

Practical implication: Re-test AI behaviour whenever prompts, tools, credentials, or downstream integrations change.

Threat narrative

Attacker objective: The objective is to make the AI system perform actions that are formally authenticated but operationally out of policy, so misuse scales through trusted automation.

1. Entry occurs when an AI system is connected to production data sources, APIs, and workflows using delegated credentials that appear legitimate.
2. Credential or entitlement abuse follows when the system can call endpoints or retrieve records beyond the minimum scope needed for the task.
3. Escalation happens through chained actions or behavioural drift, where individually valid steps combine into an unsafe business outcome.
4. Impact is unauthorized data exposure, policy violation, or unintended workflow execution at machine speed.

Breaches seen in the wild

• tj-actions/changed-files compromise 2025: A stolen bot token let attackers poison tj-actions/changed-files so pipelines printed their CI/CD secrets to public logs (CVE-2025-30066).
• Microsoft SAS token exposure 2023: An over-permissive Azure SAS token in a Microsoft AI GitHub repo exposed 38TB, including workstation backups and Teams messages, for 3 years.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime behaviour is now the security boundary for AI systems: once AI can call tools and move data, model accuracy is only one input to risk, not the control that decides whether the system is safe. The real governance test is whether the system stays inside its approved execution envelope after deployment. Practitioners should treat runtime validation as a core control objective, not a post-release comfort check.

AI security testing exposes a delegated-identity problem hiding inside an AI problem: when an agent acts through service accounts, tokens, or shared credentials, the identity layer becomes the determinant of blast radius. That makes entitlement scope, auditability, and offboarding central to AI assurance. Organisations that separate model review from identity governance will miss the path where valid behaviour becomes unauthorized action.

Behavioural drift creates a governance debt that static testing cannot retire: prompts, tools, and integrations change faster than annual reviews or release-gate testing can absorb. That means the control gap is not just missing detection, but missing re-validation after context changes. Runtime validation debt: the longer organisations rely on one-time assessment, the more they accumulate unseen access and execution risk. Practitioners should move to continuous evidence of control effectiveness.

AI security testing is converging with API security and access governance: the article’s core insight is that autonomous systems do not create a new universe of risk so much as compress existing identity and authorization failures into machine-speed execution. That is why API scope, workflow authorisation, and identity context must be assessed together. Security teams should align AI testing with the same rigor used for privileged access and sensitive API flows.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.

What this signals

Runtime evidence must replace assumption-based assurance: organisations should not confuse a passing pre-deployment test with ongoing control effectiveness. AI systems are more like dynamic service actors than static software features, so the control must follow the behaviour in production, not the release checklist.

AI security teams now need identity-aware validation: when an AI system acts through service accounts or delegated tokens, the relevant control question is whether the identity can do only what the task requires. That puts entitlement review, audit logging, and drift detection at the centre of AI security operations.

For practitioners

• Define runtime test cases for every high-risk AI workflow Cover API invocation scope, data access boundaries, and action authorisation for the exact workflows the system will run in production.
• Bind AI actions to least-privilege identities Review the service accounts, delegated tokens, and shared credentials an AI system uses, then constrain each to the minimum approved endpoints and data sets.
• Re-test after every tool or prompt change Treat new integrations, prompt revisions, credential changes, and data-source updates as triggers for fresh behavioural validation, not as routine configuration tweaks.
• Instrument production for runtime visibility Log the APIs called, data accessed, and actions executed so teams can correlate sequences over time and detect scope creep before it becomes an incident.

Key takeaways

• AI security testing is shifting from model evaluation to runtime assurance because autonomous systems can call APIs, access data, and execute workflows after deployment.
• Delegated identities make AI systems an IAM and NHI governance issue as well as a model-risk issue, because the identity determines the blast radius.
• Continuous validation is the practical control gap to close, since one-time testing cannot keep pace with prompt changes, new tools, and behavioural drift.

Key terms

• Offensive AI Security Testing: Offensive AI security testing is adversarial validation of AI systems and agents to expose unsafe behavior before attackers can exploit it. It goes beyond simple prompt checks and examines tools, memory, workflows, permissions, and autonomous actions to reveal realistic failure paths, misuse conditions, and control gaps.
• Behavioural Drift: Behavioural drift is the gradual change in what an identity does compared with what it was originally approved to do. For AI agents, drift can come from prompt changes, model updates, expanded integrations, or altered workflows, which makes access review alone an incomplete control.
• Delegated Identity: Delegated identity is when one actor acts on behalf of another with explicit permission and bounded authority. In AI-assisted commerce, it requires clear consent, limited scope, and traceable records so the retailer can distinguish authorised delegation from unauthorised automation.
• Runtime Visibility: The ability to observe what an AI client actually accessed, which tools it used, and how it behaved during a session. It is more useful than entitlement snapshots for agent governance because it captures executed reality, not just approved access.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management in the context of operational control. It helps practitioners connect delegated access, lifecycle governance, and auditability to the realities of autonomous systems.