AI pentesting tools expose the new attack surface in LLMs

At a glance

What this is: This is an Obsidian Security analysis of AI pentesting tools for LLMs and autonomous agents, with the key finding that conventional testing does not cover AI-specific attack paths such as prompt injection, model inversion, and memory poisoning.

Why it matters: It matters because IAM, NHI, and AI governance teams now have to assess how AI systems access data, invoke tools, and operate with permissions that can exceed their intended scope.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Obsidian Security's analysis of AI pentesting tools for LLMs and autonomous agents

Context

AI pentesting tools exist because conventional vulnerability scanners were built for software flaws, not for systems that interpret language, retain memory, and take actions on behalf of users or processes. In practice, LLMs and autonomous agents create a security gap between access control, behavioural testing, and data governance, which is where identity governance becomes relevant.

The article frames a real operational problem for enterprise AI programmes: models and agents can be tested for technical weakness, but their permissions, tool use, and downstream data access still need explicit governance. That intersection matters for NHI, IAM, and agentic AI teams because AI systems increasingly behave like privileged non-human identities with inconsistent controls.

Key questions

Q: What breaks when AI pentesting is not built into the release process?

A: Security teams lose visibility into how model changes affect behaviour, tool use, and data exposure. That means a model can pass functional testing while still becoming unsafe in production, especially after new connectors, prompts, or memory layers are added. Continuous adversarial testing catches those changes before they become a governance failure.

Q: Why do AI agents complicate existing IAM and NHI controls?

A: They complicate control design because they can select actions at runtime, call multiple APIs, and move authority across systems without a human session boundary. That breaks assumptions built into static entitlements and traditional service account management. Governance has to account for delegated action, changing context, and auditability across the full execution chain.

Q: How do teams know if AI-assisted pentesting is actually working?

A: Look for higher-quality findings, faster triage, and fewer unresolved false positives, not just more output. If the workflow still requires manual cleanup to make findings usable, the tool is adding noise rather than improving decision quality. Effective testing should shorten the path from discovery to verified action.

Q: What accountability exists when an AI agent misuses enterprise access?

A: Accountability sits with the organisation that granted the access, defined the workflow, and accepted the control gaps. Risk, security, and system owners should jointly document who approved the permissions, which controls limit the agent, and what evidence shows those controls were tested before production use.

Technical breakdown

Why traditional penetration testing misses AI attack paths

Conventional penetration testing evaluates known application and infrastructure weaknesses such as misconfiguration, injection, or exposed services. AI systems fail differently because the attack surface includes prompt interpretation, model behaviour, memory, and tool invocation. A scanner can tell you whether an endpoint responds, but not whether a crafted prompt can coerce a model into disclosing sensitive content or taking unsafe actions. That is why AI pentesting tools must simulate adversarial inputs and test decision boundaries, not just code paths.

Practical implication: security teams need AI-specific test cases and coverage criteria, not a lifted-and-shifted web app pentest checklist.

How prompt injection and model inversion change the risk model

Prompt injection manipulates the model through malicious instructions embedded in user or retrieved content, causing unsafe outputs or unauthorized actions. Model inversion goes further by attempting to reconstruct training data or sensitive attributes from model responses and internal patterns. Both threats exploit the fact that models are probabilistic systems that can be influenced by inputs rather than simply queried for static answers. For enterprises, the risk is not just incorrect output, but exposure of customer data, proprietary context, or policy-bypassing behaviour.

Practical implication: teams should test for prompt and data exfiltration paths before exposing models to sensitive internal content.

Why autonomous agents need continuous adversarial testing

Autonomous agents are harder to secure than single-shot LLM interactions because they can chain prompts, call tools, retain memory, and continue executing without constant human intervention. Memory poisoning can alter future decisions, while compromised tool access can turn a model from assistant into actuator. This is where AI security converges with NHI governance, because the agent’s permissions, identity, and runtime context become as important as its model quality. Continuous red-teaming and integration into MLOps are therefore essential, not optional.

Practical implication: organisations should test agents throughout the lifecycle, including after model updates, new connectors, and permission changes.

Threat narrative

Attacker objective: The attacker wants to turn AI behaviour into a control bypass that leaks data, misuses tools, or produces unsafe downstream actions.

1. Entry occurs when an attacker sends crafted prompts, injects malicious retrieved content, or abuses exposed AI endpoints to influence model behaviour.
2. Escalation happens when the model is manipulated into revealing sensitive information, calling tools it should not use, or persisting poisoned memory that changes later decisions.
3. Impact follows when the agent or model exposes data, performs unsafe actions, or amplifies the attacker’s access into broader enterprise systems.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI pentesting is now a governance function, not just a technical test. The article shows that model robustness, tool abuse, and memory poisoning cannot be left to isolated red teams or point security reviews. In identity terms, the issue is not only whether the model works, but what it is allowed to access and execute. Enterprises should treat AI security testing as part of broader access governance, with clear ownership across IAM, security, and MLOps.

Agent identity is becoming a first-class control problem. Once an AI system can invoke tools, query internal data, or act across workflows, it behaves like a non-human identity with runtime privileges. That creates a distinct governance burden because the model is not just software, it is a decision-making entity with access scope that can drift. The named concept here is agent access drift: the widening gap between intended permissions and the permissions an agent can effectively exercise. Practitioners should govern agent identities as dynamic identities, not static integrations.

Specialised AI pentesting should be evaluated against the full lifecycle of AI risk. Testing only the model in isolation misses the real enterprise exposure, which emerges when prompts, retrieval layers, memory, connectors, and deployment pipelines intersect. That makes AI security a control-plane problem as much as a model problem. The more connected the environment becomes, the more important it is to test the whole chain, not just the prompt surface.

Compliance evidence will increasingly depend on demonstrable adversarial testing. The article correctly points to audit trails, coverage, and remediation timing as metrics that matter to governance teams. That is where AI security converges with NIST AI RMF expectations around measurement and management, and with OWASP agentic guidance on tool abuse and prompt manipulation. Organisations that can prove continuous testing will be better placed to defend AI deployment decisions to risk and compliance stakeholders.

AI pentesting tools are also exposing a control gap in SaaS and cloud access governance. Once AI systems connect to enterprise applications, the real question becomes whether access is scoped, reviewed, and revocable at the same pace as model change. That is an NHI problem as much as an AI problem. Security teams should assume every connected agent can become a privilege amplifier unless runtime access is continuously constrained.

From our research:

• 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments, according to AI Agents: The New Attack Surface.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• See also: our OWASP Agentic AI Top 10 guide for a broader control map across prompt injection, tool misuse, and agent trust boundaries.

What this signals

Agent access drift will become a recurring governance issue as organisations connect more AI systems to business workflows. The practical challenge is not whether agents can be deployed, but whether their access scope can be constrained, reviewed, and removed with the same discipline used for privileged human access. For teams aligning to NIST AI Risk Management Framework, that means treating runtime permissions as a measurable risk signal, not an implementation detail.

The next stage of AI security programmes will blend adversarial testing with identity governance. Once agents can act across applications, the control objective shifts toward proving that each tool call, data access, and external action is authorised for that exact runtime context. That is why OWASP Agentic AI Top 10 style risk mapping matters for practitioners planning production rollout.

AI security teams should expect audit requests to move from model accuracy to access evidence. The useful questions will be whether the organisation can show who approved the agent, what it could reach, and how quickly that access can be withdrawn when behaviour changes.

For practitioners

• Test prompt and retrieval abuse paths Create adversarial test cases for prompt injection, malicious retrieval content, and indirect instruction chains before any model touches sensitive datasets or internal workflows.
• Inventory AI-connected privileges Map every model, agent, connector, and API token to the data and actions it can reach, then revoke anything that is not strictly required for the use case.
• Add adversarial testing to MLOps gates Trigger security assessments when models, prompts, retrievers, or tool permissions change, and block release if coverage does not include runtime behaviour.
• Separate observation from execution Require higher assurance for agents that can write data, initiate transactions, or call privileged tools than for read-only assistants, especially in production.
• Preserve audit evidence for AI decisions Retain prompt, response, tool-call, and permission logs so compliance teams can reconstruct what the model accessed and why a security finding was raised.

Key takeaways

• AI pentesting now addresses behaviour, memory, and tool use, not just classic application flaws.
• The scale problem is governance blind spots, because many organisations still cannot fully audit what their AI agents access.
• Practical defence depends on continuous adversarial testing plus identity-aware access control for every connected agent.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Inversion: Model inversion is an attack technique that tries to recover sensitive information from a model by querying it in a targeted way. The risk matters because information hidden in training data, prompts, or internal behaviour can sometimes be inferred from outputs rather than directly accessed.
• Memory Poisoning (ASI06): An attack where malicious content is injected into an AI agent's memory or context, causing it to alter its behaviour in subsequent tasks — potentially exfiltrating secrets, escalating privileges, or acting against its intended purpose.
• AI pentesting: AI pentesting is the use of autonomous or semi-autonomous systems to identify, validate, and report security weaknesses in software or infrastructure. In practice, the value depends on whether the system can discover real assets, produce reproducible evidence, and support repeatable operational workflows rather than just generating vulnerability labels.

What's in the full article

Obsidian Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Tool-by-tool comparison of commercial and open-source AI pentesting platforms for LLM and agent assessments
• Implementation considerations for integrating AI security testing into MLOps and release gates
• Examples of vulnerability metrics, coverage metrics, and remediation timing used to judge programme maturity
• Guidance on how AI pentesting findings can be folded into broader SaaS and access governance workflows

👉 The full Obsidian Security post covers tool categories, workflow integration, and evaluation criteria in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It gives security and IAM practitioners a common control language for governing non-human access across modern programmes.