How should security teams scan AI agents for prompt injection and unsafe tool use in production environments?

Why This Matters for Security Teams

Prompt injection and unsafe tool use turn an AI agent into a live execution path, not just a chat interface. A scanner that only looks for jailbreak phrases misses the real risk: the agent may chain tools, retrieve sensitive context, or act on hidden instructions embedded in documents, tickets, emails, or web content. That is why agent scanning has to inspect prompts, tool schemas, memory, and downstream effects together, as reflected in the OWASP Agentic AI Top 10 and NHI research on OWASP NHI Top 10.

The operational issue is that agents do not behave like static applications. They make decisions at runtime, call tools conditionally, and can amplify one malicious input into multiple unsafe actions. The practical consequence is that production scanning has to validate whether an agent can be induced to reveal secrets, overreach permissions, or perform destructive actions through a legitimate workflow. Guidance from the NIST AI Risk Management Framework and CSA MAESTRO agentic AI threat modeling framework both point toward continuous, context-aware evaluation rather than one-time prompt checks. In practice, many security teams encounter unsafe tool use only after an agent has already written, deleted, or exposed data through a trusted integration.

How It Works in Practice

Effective production scanning should simulate adversarial interaction with the agent’s actual runtime path, not a toy prompt loop. Start by cataloging the agent’s tools, memory stores, connectors, and escalation points, then run test cases that attempt indirect prompt injection, instruction hierarchy confusion, tool misuse, and data exfiltration. Pair those tests with code-aware analysis that traces where untrusted content enters the agent, how it is transformed, and which tools can be triggered from that path. NHI incident patterns such as Gemini AI Breach — Google Calendar Prompt Injection and Amazon Q AI Coding Agent Compromised show why hidden instructions and tool access must be tested together.

Strong programs usually combine three layers:

• Pre-deployment adversarial testing for prompt injection, tool abuse, and sensitive output leakage.
• Runtime policy checks that block high-risk tool calls unless the context, user intent, and data classification all align.
• Post-action telemetry that records which tool was called, what data was accessed, and whether the action matched policy.

Where possible, scan the agent’s code and orchestration logic so the test can follow conditional branches, retries, and fallback behaviors. This is especially important when the agent can call external APIs, generate code, or write to production systems. The most useful findings are usually not “the prompt was bad,” but “the agent had a valid path to do the wrong thing.” These controls tend to break down when the agent is allowed to inspect broad context, reuse long-lived credentials, and call multiple tools without request-time authorization checks.

Common Variations and Edge Cases

Tighter scanning often increases test volume and operational overhead, requiring organisations to balance coverage against deployment speed. That tradeoff is especially visible in production systems where agents act across SaaS apps, internal APIs, and developer tools, because each connector creates a new injection and tool-abuse surface. Current guidance suggests prioritising the highest-impact paths first: privileged write actions, secret-bearing workflows, external retrieval, and any tool that can move data outside the trust boundary.

There is no universal standard for this yet, but best practice is evolving toward scenario-based scanning that mirrors real business tasks. For example, an agent that summarizes support tickets should be tested for hidden instructions in ticket text, while a coding agent should be tested for malicious code comments, poisoned package metadata, and unsafe repository commands. The Analysis of Claude Code Security and Replit AI Tool Database Deletion both underscore that the risky behavior often appears in tool execution, not in the model response itself. Security teams should also track exposure with research such as AI Agents: The New Attack Surface report, which highlights how often agent actions exceed intended scope. In production environments, the hardest cases are agents with persistent memory, broad retrieval access, and write permissions to business-critical systems.

Related resources from NHI Mgmt Group

• How should security teams handle tool discovery for AI agents in MCP environments?
• How should security teams reduce prompt injection risk in AI agents?
• How should security teams govern AI training pipelines that use agents and synthetic environments?
• How should security teams implement authorization controls for AI agent tool calls in production environments?