What should organisations do when AI testing finds unsafe tool behaviour?

Why This Matters for Security Teams

Unsafe tool behaviour is rarely just a model-quality issue. When an AI system can call tools, create tickets, query data, or trigger workflows, a weak test result exposes privilege, approvals, logging, and connector governance. That makes the problem operational: the same failure path can be repeated unless access is reduced and the action chain is constrained. Current guidance suggests treating these findings as part of AI risk management, not as isolated prompt tuning, as reflected in the NIST AI 600-1 Generative AI Profile.

Security teams often miss that tool use creates a live control plane around the model. If the AI can reach sensitive systems, a single unsafe action can become data exposure, unauthorised change, or fraudulent transaction support. The right response is to reduce what the agent can do, define when human approval is required, and ensure each tool invocation is attributable and reviewable. In practice, many security teams encounter the real weakness only after an unsafe action has already been executed, rather than through intentional red-team testing.

How It Works in Practice

The practical response starts with scoping the AI system like any other privileged workflow. First, identify which tools are genuinely required for the use case. Remove unused connectors, block broad search or write permissions, and separate read-only from change-capable actions. Then shorten token lifetimes so a compromised session or misrouted command cannot persist for long. Where the AI can initiate high-impact actions, introduce approval gates for specific thresholds, such as payments, data exports, production changes, or account modifications.

Testing should also distinguish between model behaviour and orchestration behaviour. A system may produce a bad recommendation, but the control failure occurs when the agent is allowed to act on it without validation. That is why organisations should validate output before execution, restrict tool parameters, and log the full chain from prompt to tool call to outcome. OWASP guidance for agentic systems and NIST AI governance both point toward tightening the action surface, not just improving the model response. For AI systems with autonomous tool use, that means policy checks, step-up review, and explicit deny rules for sensitive operations.

• Map every tool to a business purpose and a risk tier.
• Require human approval for destructive, irreversible, or externally visible actions.
• Use short-lived credentials and rotate any secrets used by the agent.
• Log prompts, tool calls, approvals, and final actions for audit and incident response.
• Retest after remediation to confirm the unsafe action is no longer reachable.

Where this guidance breaks down is in loosely governed environments with shared API keys, ad hoc connectors, and no central inventory, because no one can prove which tool path the agent can still reach.

Common Variations and Edge Cases

Tighter tool controls often increase workflow friction, requiring organisations to balance response speed against safety and auditability. That tradeoff becomes sharper in customer-facing or high-volume environments where every approval adds delay. Best practice is evolving, but current guidance suggests reserving the highest friction controls for the most consequential actions, while using lighter validation for low-risk retrieval or summarisation tasks.

Edge cases appear when the AI is embedded in multi-step automation. A harmless first action can become unsafe after it passes through downstream systems, so testing must cover the full workflow, not only the model response. Another common exception is sandboxed experimentation: development environments may tolerate broader tool access, but those controls must be clearly separated from production. Organisations should also be careful with temporary exceptions. A one-time connector exception can become permanent if there is no expiry and no review.

For risk teams, the key question is whether the unsafe behaviour is repeatable under the same conditions. If it is, treat it like an access-control defect and document the exact control that blocked it. If the behaviour only appears with unusual prompts, weak data, or uncommon tool combinations, that still warrants remediation, but the fix may be a combination of prompt hardening, permission reduction, and monitoring rather than a single model update. This is where agent governance intersects with IAM, because tool access is effectively identity with execution authority.

Related resources from NHI Mgmt Group

• Should organisations invest in AI offensive testing before adversaries do?
• How can organisations reduce blast radius when an AI tool is compromised?
• Should organisations prioritise tool scoping or skill governance first for AI agents?
• How can organisations reduce unsafe AI outputs without over-restricting users?