How should security teams detect malicious AI tool calls without relying only on logs?

Why This Matters for Security Teams

Detecting malicious AI tool calls is not the same as reviewing application logs after the fact. Tool use can convert a prompt injection, poisoned retrieval result, or compromised agent into an action that changes data, triggers workflows, or exposes secrets. Security teams need more than event records because the risky part is often the argument set, the source of the input, and whether the action matches the agent’s expected purpose.

This is a control problem, not just a visibility problem. A model can appear to behave normally while still issuing a harmful write request, using an oversized parameter, or forwarding sensitive content into a downstream system. Current guidance from the NIST Cybersecurity Framework 2.0 supports broader detection and response capabilities that go beyond simple logging, especially where operational resilience depends on knowing what a system tried to do, not only what it said. In practice, many security teams discover malicious tool use only after a workflow has already executed or a data store has already been modified, rather than through intentional detection design.

How It Works in Practice

Effective detection starts by treating every tool call as a structured security event. Teams should capture the tool name, arguments, payload size, source context, user or agent identity, and the decision path that led to the call. That gives defenders enough material to compare each action against an expected baseline for the specific agent, workflow, and privilege level.

The strongest pattern is to combine several checks rather than depend on a single alert condition:

• Validate whether the tool call matches the task the agent was assigned.
• Compare argument ranges, formats, and destinations against normal behaviour.
• Flag write actions from workflows intended to read, summarise, or classify only.
• Inspect whether retrieved content, web content, or user input influenced the call.
• Preserve provenance so investigators can see which source introduced the risky instruction.

For AI-specific threat modelling, MITRE ATLAS is useful for understanding adversarial techniques that influence model behaviour and action selection. For agentic environments, the relevant question is not just whether the call succeeded, but whether the agent’s tool choice was manipulated by prompt injection, indirect instruction, or contaminated context. Where an agent has write authority, teams should also require policy checks before execution, not after the event has been committed.

Operationally, this works best when detections are correlated with business logic, identity context, and downstream side effects. If a support agent suddenly starts calling an admin-only export tool, or a summarisation workflow begins issuing payment or deletion actions, the anomaly should be visible even if the raw log entry looks syntactically valid. In environments with high call volume, limited metadata, or weak tool schemas, these controls tend to break down when many tools share similar names because normalisation makes malicious calls look routine.

Common Variations and Edge Cases

Tighter tool-call inspection often increases latency and operational overhead, requiring organisations to balance response speed against stronger abuse detection. That tradeoff becomes sharper in autonomous agents, where blocking too aggressively can interrupt legitimate task completion.

There is no universal standard for this yet. Some teams rely on schema validation plus allowlisted tool paths, while others add behavioural baselines or human approval for high-risk actions. The right model depends on whether the agent is read-only, semi-autonomous, or allowed to execute privileged changes. For higher-risk systems, the OWASP Top 10 for LLM Applications is a practical reference for prompt injection, tool misuse, and output handling risks.

Edge cases matter. A small call can still be malicious if it targets a privileged endpoint, while a large payload may be harmless if it is part of a known export job. Likewise, tool calls driven by RAG content need special scrutiny because the retrieved source may be untrusted even when the agent appears confident. Teams should also account for read-receive-write chains, where a model ingests hostile instructions from one system and passes them into another without ever exposing the malicious content in a human-readable log.

For governance and control mapping, CISA Secure by Design reinforces the idea that security should be built into the workflow boundary itself. The practical takeaway is to detect not only what the agent said, but whether the call was appropriate, permitted, and provenance-backed for that exact context.

Related resources from NHI Mgmt Group

• How should security teams detect AI-written malware without relying on signatures?
• How should security teams govern AI agent tool calls without exposing credentials?
• How should security teams detect agentic AI usage without relying only on EDR?
• How should security teams monitor AI agents without relying on sampled logs?