AI security testing is becoming a core control for models and agents

At a glance

What this is: The article argues that AI security testing needs specialised methods because traditional penetration testing misses model- and agent-specific attack paths.

Why it matters: That matters to IAM practitioners because AI agents, delegated tools, and privileged integrations introduce identity and access risks that conventional testing does not adequately govern.

By the numbers:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

👉 Read Obsidian Security's analysis of AI security testing for models and agents

Context

AI security testing has emerged because conventional application testing was built for predictable code paths, not systems that generate outputs, retain memory, and invoke tools at runtime. In practice, that creates a governance gap across model behaviour, delegated access, and the systems those agents can touch, including IAM and NHI controls that were never designed for autonomous decision loops.

The article focuses on the mismatch between AI deployment speed and security validation maturity. For practitioners responsible for identity, this means the access model around agents, service accounts, APIs, and data stores has become part of the attack surface, not just the application layer.

Obsidian Security positions continuous validation as the operational answer, but the deeper issue is that AI testing must now connect technical exploitability to governance, audit, and remediation workflows. That starting point is becoming typical for enterprises adopting agents; the controls, however, are still atypical.

Key questions

Q: How should security teams implement AI evaluation in production workflows?

A: Security teams should treat AI evaluation as a continuous control, not a pre-launch checklist. Build representative datasets, define scoring criteria for the outcomes that matter, and rerun tests whenever prompts, models, or retrieval logic change. That creates evidence for regression detection and release decisions instead of relying on intuition.

Q: Why do AI systems create identity and access risk beyond traditional AppSec?

A: Because AI systems often act through delegated access. When a model can use tools, retrieve data, or trigger actions, it becomes a runtime decision-maker with privileges that can be misused through prompt injection, poisoned context, or overbroad permissions. That is an identity problem as much as a code problem.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: Who should own AI security testing findings when agents are connected to business systems?

A: Ownership should sit with the team that can actually change the exposed access path, usually a combination of MLOps, security, and the system owner for the connected tool or data source. If no one owns the agent's permissions, the organisation will record findings without shrinking blast radius or closing the privilege gap.

Technical breakdown

Why traditional penetration testing misses AI attack surfaces

Traditional penetration testing assumes systems fail through known classes of network, code, or authentication weakness. AI systems fail differently because the model itself can be manipulated through prompt injection, adversarial inputs, membership inference, and output shaping. Those attacks target the reasoning path, the training data, and the tool chain the model can call. In agentic systems, the failure is often not a crash but a bad decision that looks legitimate. That is why AI security testing has to examine context windows, memory, and downstream actions, not just application endpoints.

Practical implication: build AI-specific test cases that validate behaviour, tool use, and data exposure, not just infrastructure and API security.

How memory and tool misuse create agentic AI risk

Agentic systems become more dangerous when persistent memory and integrated tools are available. Memory poisoning can seed malicious instructions that survive across sessions, while tool misuse can persuade an agent to query databases, send data, or trigger actions outside intended scope. The security issue is not only whether the model can answer correctly, but whether it can be coerced into carrying out privileged operations. This is where AI governance intersects directly with IAM and NHI, because access rights assigned to the agent determine how far a manipulation can travel.

Practical implication: scope agent permissions tightly and test for misuse paths to every API, database, and external service the agent can reach.

Why continuous testing must be tied to MLOps and governance

AI security testing cannot be a one-time gate if models and agents change continuously. New prompts, fine-tunes, tool integrations, and workflow updates can reintroduce exploitable behaviour after release. Continuous testing in MLOps pipelines is therefore a control function, not a quality function. It needs to generate evidence for risk owners, feed incident workflows, and support remediation tracking. Without that link, testing becomes a set of isolated findings rather than a governance mechanism that changes exposure over time.

Practical implication: connect AI test results to risk registers, release gates, and remediation ownership before agents move into production.

Threat narrative

Attacker objective: The attacker wants to convert a trusted AI system into a decision and access amplifier that exposes data or performs unauthorised actions.

1. Entry occurs when an attacker crafts adversarial prompts, malformed inputs, or poisoned context that the model or agent accepts as legitimate.
2. Escalation follows when the system uses its own memory, integrated tools, or delegated permissions to act on the malicious instruction.
3. Impact occurs when the agent leaks sensitive data, executes inappropriate actions, or corrupts downstream business decisions at scale.

NHI Mgmt Group analysis

AI security testing is now an access control problem, not only a model safety problem. Once an agent can call tools, read memory, or access data stores, the test surface becomes an identity surface. That means security teams must judge whether an exploit path can expand rights, not just whether the model behaves incorrectly. The practical conclusion is that AI security testing belongs in governance discussions alongside IAM, PAM, and NHI lifecycle controls.

Continuous validation is the right pattern because AI risk changes after deployment. Prompts, connectors, permissions, and workflows all change faster than most annual or quarterly review cycles. That makes static assessment inadequate for agentic systems. Practitioners should treat continuous testing as a control loop tied to release gates, incident response, and risk acceptance.

Context poisoning is the named concept security teams should watch closely. It captures the way malicious instructions can survive in prompts, memory, or conversation history and then influence later actions. This is especially relevant where agents have persistent state and business tool access. The practical conclusion is that teams need explicit controls over what context an agent can retain and trust.

AI testing metrics need to shift from coverage rhetoric to operational evidence. Vulnerability coverage, remediation time, and attack success rate are more meaningful than generic assurance statements because they show whether the organisation can actually detect and contain agent-specific weakness. That makes testing part of risk management, not a badge of maturity.

Identity governance becomes the constraint on agentic blast radius. The decisive question is no longer whether an AI system can reason, but which identities it can assume, what tokens it can mint, and how quickly those rights can be revoked. For practitioners, the lesson is to govern the agent as a credentialed runtime entity.

What this signals

Context poisoning: as agents retain memory and consume tools, the most important programme-level question becomes whether the organisation can bound what the agent remembers, what it trusts, and what it can do. That control posture should be measured against NIST AI Risk Management Framework principles and the AI threat patterns catalogued in the MITRE ATLAS adversarial AI threat matrix.

AI security teams should expect testing to move closer to identity governance as agents become persistent enterprise actors. The practical consequence is stronger linkage between agent inventory, delegated access, and revocation workflows, because a tested model is still unsafe if its runtime identities remain over-privileged.

The next maturity step is not more testing volume alone, but tighter coupling between findings, risk acceptance, and control enforcement. Organisations that cannot prove where agents have access will struggle to prove their security testing is reducing exposure.

For practitioners

• Map every AI agent to its real identity surface Inventory the service accounts, API keys, tokens, and delegated roles each agent can use. Treat those credentials as part of the model's attack surface and review them with the same discipline used for privileged human access.
• Add adversarial tests to MLOps release gates Run prompt injection, memory poisoning, and tool misuse tests before production promotion, then repeat them after any connector, prompt, or permission change. Block release when the agent can reach data or actions outside intended scope.
• Tie security findings to risk ownership and remediation Feed AI test results into risk registers, incident workflows, and remediation tracking so findings do not remain isolated in security tooling. Assign ownership for every failure path that exposes data, executes actions, or expands privilege.
• Constrain tool access with task-scoped permissions Limit each agent to the minimum tools, datasets, and action types required for its job, and revoke unused access as workflows change. This reduces the impact of tool misuse when an adversarial input succeeds.

Key takeaways

• AI security testing is now a governance control because agents can be manipulated into using legitimate access in illegitimate ways.
• The evidence points to a widening exposure gap, with agent deployments growing faster than the controls used to audit and constrain them.
• Practitioners should tie adversarial testing to identity governance, permission scoping, and remediation ownership or the findings will not reduce risk.

Key terms

• AI-driven security testing: AI-driven security testing uses automated analysis to discover vulnerabilities earlier in the software lifecycle. The practical value is not just speed, but earlier feedback while code, ownership, and context are still visible. It becomes useful when teams can repeat it continuously without creating extra operational friction.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool Misuse: Tool misuse occurs when an agent uses an allowed integration in a way that exceeds its intended task, scope, or risk tolerance. The problem is often not access alone but the combination of valid credentials, broad permissions, and unbounded action sequencing.
• Context Poisoning: Context poisoning is the manipulation of information that an AI agent reads before acting. The malicious content does not need to be code. If it changes the agent’s instructions, tool choices, or assumptions, it can alter behaviour and expand the impact of a compromised delivery path.

What's in the full article

Obsidian Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step AI security testing workflows for development, staging, and production gates
• Tool category comparisons across open source, commercial, and cloud-native AI testing options
• Example red-team test cases for prompt injection, memory exploitation, and tool misuse
• Integration patterns for CI/CD and MLOps pipelines, including governance and audit handoff

👉 The full Obsidian Security post covers testing workflows, framework choices, and MLOps integration detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management in the context of real enterprise control gaps. It gives security and identity practitioners a common way to govern delegated access, runtime credentials, and identity lifecycle risk.