AI penetration testing needs a different methodology from web testing

At a glance

What this is: Repello AI says AI penetration testing needs its own methodology because the attack surface is model behaviour, tools, retrieval, and memory rather than code alone.

Why it matters: For IAM and NHI practitioners, the key issue is that AI systems can be steered into using elevated tools or exposed context in ways traditional testing will not catch.

Context

AI penetration testing is the discipline of testing an AI system for ways an attacker can manipulate its behaviour, not just its code. That matters because LLMs, chatbots, and agents combine prompts, tools, retrieval, memory, and output channels into a control surface that behaves differently from conventional applications.

The governance gap is that conventional penetration testing assumes deterministic logic and stable endpoints, while AI systems can change behaviour across turns, contexts, and modalities. Where the article connects to identity, the issue becomes even sharper: an AI agent with access to identity-aware agentic controls can become a new path for privilege misuse, data exposure, or delegated abuse.

Key questions

Q: How should security teams test generative AI systems for real-world abuse?

A: They should test the full execution path, not just the model prompt. That means probing retrieval, tool use, output handling, and any delegated actions the system can perform. The goal is to reproduce how an attacker would move from manipulation to impact, then capture whether the system leaks data, ignores guardrails, or performs unauthorised actions.

Q: Why do AI agents create a different risk model than chatbots?

A: AI agents can act, not just generate. Once they can browse, retrieve, call tools, and complete tasks, the main risk shifts from unsafe output to unsafe action. That expands the attack surface from prompts to workflows, credentials, and downstream systems, which is why traditional model-only controls are no longer enough.

Q: What are the signs that an AI application has too much authority?

A: A common sign is that a single model response can write to databases, send messages, start workflows, or spend resources without a separate check. Another indicator is when the same identity can reach multiple systems even though the task only needs one narrow permission set.

Q: Should organisations combine AI red teaming with IAM and PAM reviews?

A: Yes, because many of the highest-risk AI findings are really delegation problems. If the system can call tools or access data, its permissions, approvals, logging, and revocation model should be reviewed alongside the model itself. Otherwise teams may secure the prompt layer while leaving the action layer overexposed.

Technical breakdown

Why AI applications need a different test model

Traditional application testing looks for code defects such as injection, broken access checks, or unsafe parsing. AI applications are probabilistic systems whose behaviour emerges from prompts, model weights, retrieved context, tool configuration, and conversation history. That means the tester is not just probing for a bug trigger. They are probing for a way to influence decisions, tool calls, and generated outputs. In practice, the control surface includes instructions, memory, and external data sources, so a finding can exist even when no conventional code flaw is present. The right mental model is therefore not "find the broken line of code" but "find the path by which untrusted input changes what the system decides to do."

Practical implication: build a test scope that includes prompts, retrieval, tools, memory, and outputs, not only application code.

How reconnaissance maps the AI attack surface

Reconnaissance for AI systems starts by enumerating the elements that shape behaviour: system prompts, tool inventory, input modalities, state retention, retrieval sources, and output destinations. Each one changes what an attacker can attempt and how far a successful manipulation can spread. A chatbot that only renders text has a much smaller blast radius than an agent that can send email, query databases, or write files. The point of recon is to identify trust boundaries and escalation paths before active testing begins. Without that map, teams tend to overtest obvious prompt tricks and miss the higher-value paths created by tools, retrieved content, or persistent state. Read Threat Modelling AI Agents.

Practical implication: enumerate tools, retrieval sources, and stateful behaviours before running attack payloads.

Where AI pentests find the highest-risk failures

The most consequential AI test tracks are prompt injection, system prompt extraction, tool abuse, guardrail bypass, data exfiltration, and output integrity failures. These are not just content problems. They show whether the model can be persuaded to ignore instructions, reveal sensitive context, misuse tools, or surface data it should not expose. Tool abuse is often the highest-impact category because it turns language manipulation into external action. That is where AI security becomes an identity and access problem as much as a model problem, especially when the system can act with permissions broader than the human user who triggered it. See the Agentic AI Security Guide.

Practical implication: prioritise tool-enabled abuse paths before lower-impact content or style issues.

Threat narrative

Attacker objective: The attacker wants to turn AI instruction-following into unauthorized tool use, disclosure, or workflow execution.

1. Entry occurs through direct or indirect prompt manipulation, including retrieved content, uploaded files, or multi-turn conversation design.
2. Credential and privilege misuse follow when the AI is persuaded to call tools, reuse context, or act through permissions broader than the initiating user intended.
3. Impact occurs when the system leaks data, performs unintended actions, or executes an external workflow such as email, database, or file operations.

Breaches seen in the wild

• Meta AI Instagram Account Takeover: 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI pentesting exposes an identity problem as much as a model problem: once an AI system can call tools, it starts behaving like a non-human actor with delegated authority. That shifts the question from whether the model is clever to whether its permissions, context, and output boundaries are governed tightly enough. Where organisations blur user intent with agent authority, abuse becomes easier to stage and harder to contain. Practitioners should treat the agent as an identity-bearing control point, not a passive interface.

Blast radius is the right severity lens for AI systems: a small chatbot and a tool-enabled agent may both use the same model family, but their risk profile is radically different. The article is right to prioritise what the AI can reach, not just what it can say. That is the same governance instinct used in IAM and PAM, where scope matters more than labels. Practitioners should rank test cases by reachable action, not by novelty of prompt.

Tool abuse is the named concept practitioners should adopt: the decisive failure mode is not a clever prompt on its own, but the combination of language manipulation and over-broad tool access. Once the AI can chain tools, a single successful influence step can become data movement, fraud, or lateral action across connected systems. That makes permission design, parameter validation, and action logging central to AI governance. Practitioners should test the tool boundary as aggressively as the model boundary.

Persistent context creates hidden security debt: memory and retrieval make AI behaviour sticky across sessions, which means a weak decision in one turn can shape later outcomes long after the original input has gone. That creates a governance gap that conventional point-in-time testing often misses. The issue is not just leakage but durable influence. Practitioners should govern retained context as an attack surface, not as a convenience feature.

AI security testing now belongs in the same control conversation as IAM and PAM: when an agent can query systems, send messages, or write records, it is no longer just a content generator. It is an acting system with delegated reach. That means access scope, approval boundaries, and logging discipline should be designed together. Practitioners should align AI testing with identity controls before the first production workflow goes live.

Read next: Agentic AI Security Guide

What this signals

AI security governance now has to cover delegated action, not just generated content: once a model can call tools, retained context and external services become part of the control boundary. The practical shift is from content moderation to permission design, logging discipline, and containment of action chains. Practitioners should treat each enabled tool as a security decision, not a convenience feature.

Context persistence is becoming a hidden control gap: conversation history, retrieved documents, and memory can all carry attacker influence forward across sessions. That means a single successful manipulation can have effects that outlast the original prompt. Teams should design detection and review around stateful influence, not only visible malicious prompts.

For practitioners

• Map AI trust boundaries before testing Inventory system prompts, tools, retrieval sources, memory, input modalities, and output channels so testers can target the real escalation paths. Read Threat Modelling AI Agents.
• Prioritise tool-enabled abuse scenarios Put send_email, database, file-write, and external request tools ahead of lower-impact prompts because those paths turn influence into action.
• Test indirect prompt injection explicitly Include uploaded files, retrieved documents, and tool responses in your test plan because attacker instructions often arrive through trusted content.
• Require reproducibility before severity is assigned Run each successful attack sequence multiple times and record the exact transcript, model version, and configuration so non-deterministic behaviour is judged consistently.
• Align AI permissions with delegated identity controls Restrict the agent to the minimum tool scope it needs in each workflow and separate user intent from agent authority wherever external actions are possible.

Key takeaways

• AI penetration testing is a different discipline from web testing because the attack surface includes prompts, tools, retrieval, memory, and output behaviour.
• The highest-risk failures are tool abuse, prompt injection, system prompt extraction, and data leakage through model-mediated workflows.
• Practical testing must focus on delegated authority and blast radius, because that is where language manipulation becomes operational impact.

Related NHI Mgmt Group resources: Agentic AI Security Guide · Threat Modelling AI Agents · Threat Modelling Guide

Key terms

• AI Penetration Testing: AI penetration testing is the practice of using artificial intelligence to help find security weaknesses in systems, applications, and workflows. It applies automated reasoning, pattern detection, and test generation to identify attack paths, misconfigurations, and control gaps, while still requiring human oversight to validate findings and interpret risk.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool Abuse: Tool abuse occurs when an AI assistant is manipulated into making actions through connected APIs, plugins, or workflows that the user did not intend. The risk is not only incorrect output, but execution of legitimate systems with attacker-chosen inputs or destinations.
• Trust Boundary: A trust boundary is the point where one system’s authority should stop and another system’s authority should begin. For internal automation, weak trust boundaries let monitoring, remediation, and execution share privileges that should have remained separate.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the delegated access patterns that modern AI systems introduce.