Because the root causes differ. Wrong-tool errors usually point to planning, routing, or intent recognition problems, while right-tool, bad-argument errors point to grounding, parameter extraction, or context-handling issues. If you do not split them, you cannot tell whether to change the prompt, the evaluator, the schema, or the surrounding tool design.
Why tool-calling failures need separate measurement
Tool-calling failures are not one problem. A model that chooses the wrong tool is failing earlier in the chain than a model that selects the right tool but passes the wrong arguments. Those failures come from different root causes, so they need separate measurement if you want to know whether the fix belongs in planning, routing, schema design, prompting, or tool interface design.
What each failure mode is actually telling you
The most useful split is between tool selection and tool execution. Wrong-tool errors usually indicate a planning, intent recognition, or routing problem, where the system did not identify the right capability at all. Right-tool, bad-argument errors usually indicate grounding, extraction, or context handling problems, where the system understood the tool but failed to populate it correctly.
That distinction matters because the same aggregate success rate can hide opposite failure patterns. A system can look acceptable overall while still being unreliable in production if it frequently reaches the correct tool but supplies malformed, incomplete, or misbound parameters. Measuring the two cases separately shows whether the issue is choice, formatting, or both.
For agentic systems, that split also makes the operational boundary clearer. Tool choice is closer to planning and policy compliance, while argument quality is closer to interface correctness and state transfer. If you do not separate them, you can easily overcorrect the wrong layer, such as tightening prompts when the real defect is schema ambiguity, or rewriting tool descriptions when the real defect is poor context extraction.
How to use the split to improve the system
Separate measurement gives you a cleaner debugging path and a more reliable evaluation harness. It lets you compare tool-selection accuracy, argument-validity rate, and end-to-end task success without collapsing them into one number that cannot localize failure.
What to verify: Track whether failures are concentrated in tool choice, field population, value normalization, or cross-turn state carryover. A good evaluator should tell you not only whether the task failed, but also whether the failure happened before the tool call, inside the call, or after execution.
- Use wrong-tool rate to test planning and routing quality.
- Use right-tool, bad-argument rate to test schema understanding and grounding.
- Use end-to-end success only as the outcome metric, not the diagnostic one.
Decision rule: If wrong-tool errors dominate, adjust tool descriptions, selection logic, or policy gating. If right-tool, bad-argument errors dominate, improve argument schemas, context formatting, validation, or state extraction. If both are high, the system likely has a broader orchestration problem rather than a single prompt defect.
Risk and Threat Considerations
In agentic systems, mismeasurement creates a control gap. Teams may believe they have improved reliability when they have only moved failure from one stage to another, leaving unsafe actions, missed tasks, or bad side effects unchanged.
Failure mechanism: Collapsing distinct failure modes into one metric hides whether the agent is failing to select the correct capability or failing to execute the chosen capability correctly. That makes remediation noisy and can let unsafe tool use persist under a superficially improved aggregate score.
Impact: The result is poorer incident diagnosis, weaker regression testing, and a higher chance of shipping agents that appear competent in demos but break under real task structure, especially when tool chains, schemas, or contexts become more complex.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF, NIST SP 800-53 Rev 5 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Wrong-tool and bad-argument failures are core tool-misuse modes in agentic systems. |
| ASI03 — Identity & Privilege Abuse | Tool calls determine what an agent is allowed to do and with which authority. | |
| Recommendation — Measure tool selection and argument validity separately to isolate tool misuse patterns. Separate authorization from execution failures so privilege issues are not hidden by task success. | ||
| NIST AI RMF | AI governance and risk management | Agentic evaluation needs risk-aware measurement, documentation, and monitoring discipline. |
| Recommendation — Define evaluation metrics that distinguish planning errors from execution errors. | ||
| NIST SP 800-53 Rev 5 | AU-2 — Event Logging | Trace-level logging is needed to distinguish wrong-tool from bad-argument failures. |
| Recommendation — Log tool choice and arguments distinctly so failures can be diagnosed from the trace. | ||
| OWASP ASVS | V4 — API and Web Service | Tool calls behave like service requests and need separate validation of target and parameters. |
| Recommendation — Validate the selected operation and its parameters as separate checks. | ||
Practitioner Guidance
What to prioritise: Treat the measurement design itself as part of the control surface. The first question is not whether the agent succeeded, but where the failure occurred in the tool-calling pipeline.
What good looks like: Your evaluation set should let you report separate rates for tool selection, argument correctness, and task completion, with examples that make the failure class obvious to an engineer reviewing the trace.
Common mistake: Do not use a single pass/fail metric and then infer the fix from intuition. That often sends teams to the wrong layer, which wastes iteration cycles and leaves the real failure mode intact.
Practitioner takeaway: Separate measurement is what turns tool-calling evaluation from a score into a diagnosis, and diagnosis is what tells you whether the system needs better planning, better grounding, or better tool design.
Related resources from NHI Mgmt Group
- How can organizations mitigate tool misuse in agentic deployments?
- What are the core risks identified by the OWASP Agentic Top 10?
- When does just-in-time access reduce risk for agentic AI, and when does it fall short?
- How should security teams govern machine identity credentials in agentic AI environments?