Topic starter
20/08/2026 6:00 pm
TL;DR: Tool-calling LLMs fail in two distinct ways, either choosing the wrong tool or choosing the right tool with bad arguments, and Arize shows these errors need separate measurement because one hides selection problems while the other exposes invocation quality. That distinction matters because agentic systems create governance and reliability risks that conventional evaluation workflows often blur.
Editorial analysis by NHI Mgmt Group, based on content published by Arize: “How to Evaluate Tool-Calling Agents”.
Key questions
Q: How should teams evaluate tool-calling agents when wrong-tool and bad-argument errors both occur?
A: Evaluate tool selection and tool invocation separately.
Q: Why do tool-calling failures require separate measurement in agentic systems?
A: Because the root causes differ.
Q: What are the signs that a tool-calling evaluator is too strict or too loose?
A: Look for disagreements between the judge and the real user intent.
Practitioner guidance
- Separate tool selection from tool invocation metrics Track whether the agent chose the right tool independently from whether it passed valid arguments, so prompt fixes do not mask payload failures.
- Calibrate semantic equivalence rules Define when values like city names, airport codes, or date formats should be treated as equivalent in your evaluator, then document that rule in the judge prompt.
- Add environment-specific constraint checks Encode use-case rules such as current-year assumptions, required fields, and allowed value ranges directly into the evaluation template.
Bottom line: Tool-calling agents fail in two separable ways, so one aggregate score is not enough to govern them well.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:38 pm
Tool-calling introduces a governance gap between choosing an action and executing it. The article shows that a model can appear accurate while failing in the selection layer or the invocation layer, which means a single success metric is structurally misleading. For agentic AI, that distinction is the difference between intent correctness and action correctness. Practitioners should design evaluation so each layer is visible on its own.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: How should security teams govern AI agents that choose tools at runtime?
A: Security teams should treat runtime tool choice as a governed access event, not a normal application call. That means task-scoped credentials, explicit approval boundaries for sensitive actions, and logs that record both the tool selected and the identity used. If the agent can change its plan, the control model must be able to change with it.
👉 Read our full editorial: Tool-calling agents fail in two distinct ways