How should security teams implement AI penetration testing for agents and models?

Start with the highest-risk workflows first, especially agents that can access SaaS data, APIs, or approval paths. Then combine adversarial input testing, tool-use abuse scenarios, and output leakage checks with CI/CD or MLOps gates. The goal is to catch both model weaknesses and the permission paths that let an attacker turn model behaviour into operational impact.

Why This Matters for Security Teams

AI penetration testing for agents and models is not the same as testing a static application. An autonomous agent can chain prompts, call tools, retrieve data, and act on partial instructions, which means a single weakness can become data exposure, unauthorised action, or privilege escalation. Guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both point to the same operational reality: testing has to include behaviour, context, and downstream permissions.

For security teams, the most important question is no longer whether a model can be tricked into a bad output. It is whether that output can be turned into a harmful action through SaaS connectors, API keys, approval workflows, or linked agents. That is why NHIMG research on the OWASP NHI Top 10 treats agent exposure as an identity and permission problem as much as a model problem. In practice, many teams discover the issue only after an agent has already touched production data or invoked a real workflow.

How It Works in Practice

Effective AI penetration testing should be structured around workflows, not just prompts. Start by mapping the agent’s highest-risk paths: data access, tool invocation, approval steps, external messaging, and any place where the model can influence an action that persists outside the chat. Then test each path with three layers of abuse: adversarial input designed to bypass instructions, tool-use abuse designed to make the agent call the wrong function or overreach permissions, and leakage checks that look for secrets, sensitive records, or hidden context being exposed in outputs.

This is where traditional appsec methods need adaptation. Model testing alone is insufficient if the agent still has broad access. Teams should pair red-team style testing with permission testing, short-lived credentials, and runtime controls so that a successful prompt attack does not automatically become a real incident. The CSA MAESTRO agentic AI threat modeling framework and MITRE ATLAS adversarial AI threat matrix are useful for classifying attack paths, while LLMjacking: How Attackers Hijack AI Using Compromised NHIs shows how quickly stolen access can be exploited once the agent environment is exposed.

• Test prompts against instruction hierarchy breaks, prompt injection, and indirect prompt injection through retrieved content.
• Test tools for excessive authority, unsafe defaults, and missing approval checks.
• Test outputs for secrets disclosure, data over-sharing, and policy violations.
• Run these tests in CI/CD or MLOps gates so regressions fail builds before deployment.

For identity and access, current best practice is evolving toward workload identity, intent-aware authorisation, and just-in-time secrets rather than static service accounts. That matters because an agent’s risk changes per task, per context, and per tool call. These controls tend to break down in highly dynamic multi-agent systems with shared memory and broad connector access because a single weak agent or reused token can propagate trust across the chain.

Common Variations and Edge Cases

Tighter agent testing often increases operational overhead, requiring organisations to balance faster release cycles against deeper validation and more runtime controls. There is no universal standard for this yet, so teams should treat some practices as current guidance rather than settled doctrine.

For example, not every model requires the same depth of testing. A summarisation model with no tool access needs a different test plan than an autonomous support agent that can read tickets, send email, and trigger refunds. Likewise, jailbreak tests alone are not enough when the real risk is connector abuse or hidden context leakage. Security teams should also account for retrieval-augmented generation, where the attack surface includes documents, vector stores, and embedded instructions, not just the model endpoint.

Edge cases appear when agents are nested, delegated, or given broad fallback permissions. In those environments, a failed test can look safe at the model layer while still being dangerous operationally because another agent can complete the action. The practical rule is to test the full action path, not just the conversation. That approach aligns with the guidance in Analysis of Claude Code Security and the broader intent of the NIST AI Risk Management Framework.

Related resources from NHI Mgmt Group

• How should security teams implement behavioural testing for AI agents in CI/CD?
• What breaks when security teams rely on content guardrails to control AI agents?
• What steps should security teams take to prevent Shadow AI risks?
• How should security teams implement short-lived credentials for AI agents?