Tool-calling agents fail in two distinct ways

At a glance

What this is: This is a tutorial on evaluating tool-calling agents, with a key finding that tool selection and tool invocation are separate failure modes that must be measured independently.

Why it matters: It matters because teams building agentic AI need controls that distinguish bad tool choice from bad parameters, or they will miss the real failure mode and misread system risk.

👉 Read Arize's tutorial on evaluating tool-calling agents

Context

Tool-calling agents create a governance problem because the model is no longer just generating text, it is deciding when to act, which tool to use, and what arguments to pass. In practice, that means evaluation has to separate selection failures from invocation failures, or teams will misdiagnose reliability issues and understate operational risk. For identity and access teams, this is the same kind of control separation that matters in privileged workflows: choosing the right entitlement is not the same as using it correctly.

The article uses a travel assistant to show how this works in a controlled setup, with single-tool, parallel-tool, and no-tool cases. That makes the lesson broadly applicable to agentic AI programmes, because the real issue is not one specific demo but the measurement discipline required when models can trigger external actions. The starting point is typical for organisations moving from prototype agents into production decision paths.

Key questions

Q: What breaks when an AI agent can call tools but tool selection and invocation are not measured separately?

A: You lose visibility into which failure actually occurred. The agent may choose the right tool but pass bad parameters, or choose the wrong tool even when its arguments are valid. Separate measurement lets teams isolate prompt issues, tool design problems, and capability gaps instead of treating every failure as one vague reliability problem.

Q: Why do tool-calling agents create governance problems for security and IAM teams?

A: Because the model is no longer only producing text. It is making action decisions that resemble delegated access, which means scope, approval, and error handling matter. If the agent can act in business systems, teams need controls for when it may act, what it may touch, and how failures are detected.

Q: How do teams know whether an agent evaluation judge is too strict or too loose?

A: Look for disagreement cases where the model’s output is operationally defensible even if it does not match the reference exactly. If the judge flags reasonable alternatives as wrong, it is too strict. If it misses malformed or hallucinated tool arguments, it is too loose. Calibration should follow business intent, not just label fidelity.

Q: Should organisations treat tool-calling agents as part of identity governance?

A: Yes, when those agents can trigger real actions in enterprise systems. At that point they behave like governed non-human identities with a limited operational scope. Identity governance should cover their permissions, review cadence, test coverage, and revocation path before the agent reaches production.

Technical breakdown

Tool selection vs tool invocation in agentic workflows

Tool-calling failures split into two control problems. Tool selection asks whether the model should call any tool at all, and if so, which one. Tool invocation asks whether the chosen tool receives grounded, complete, and valid arguments. These are different failure surfaces because a model can pick the correct tool yet still fabricate dates, omit required parameters, or substitute values that were never in the user request. In agentic systems, that distinction matters because the risk is not only wrong action, but also correct action with incorrect execution context.

Practical implication: evaluate selection and invocation separately so you can tell whether you need prompt changes, tool design changes, or stronger argument validation.

Why reference-free judges are useful for tool calling

Reference-free LLM-as-a-judge evaluators compare the conversation, available tools, and model output without requiring a labeled ground truth for every case. That makes them useful for early experimentation, where teams need fast feedback before they have perfect labels. The trade-off is calibration: the judge can be too strict or too lenient, especially when natural language requests allow more than one reasonable interpretation. For agentic AI governance, this is analogous to policy evaluation, where controls must be tested against intent as well as literal string matching.

Practical implication: use reference-free evaluators for speed, then calibrate them against domain expectations before treating scores as authoritative.

Custom evaluator logic for domain-specific constraints

Generic evaluation templates are rarely enough once tool use becomes operational. Some systems need date logic tied to the current year, location handling that accepts semantic equivalents, or parameter rules that reflect business-specific constraints. A custom evaluator lets teams encode those expectations so the judge catches what actually matters in production. This is especially important for agentic AI because tool outputs can be semantically valid but operationally wrong, and that gap is where automation risk accumulates.

Practical implication: extend the evaluator when the domain has non-negotiable constraints that a general judge would miss.

NHI Mgmt Group analysis

Tool evaluation is becoming a governance control, not just a model-quality exercise. Once an LLM can call tools, the security question shifts from output quality to action quality. That means organisations need to know whether the model chose the right tool, used it correctly, and stayed within the intended task boundary. In identity terms, the agent is functioning like a delegated non-human identity, so evaluation becomes part of authorization governance, not merely testing. Practitioners should treat tool evaluation as a control plane for agentic AI.

Separate measurement is the named concept this tutorial makes unavoidable. The article shows that tool selection and tool invocation fail differently, which is exactly why a single score obscures risk. A model can look strong on one dimension while still issuing malformed or hallucinated tool requests. That is a classic governance blind spot in automation, because one aggregate metric hides whether the failure is policy, prompt, or capability. Practitioners should adopt separate scoring for action choice and action execution.

Reference-free evaluation is useful, but only when teams are willing to calibrate it. The article’s CDG Airport example shows that a judge can mistake semantic equivalence for error if the rubric is too rigid. That matters because over-strict evaluation can drive unnecessary prompt churn, while over-loose evaluation can mask real failure. For NHI and agentic AI governance, the lesson is to encode operational intent, then review disagreement cases as control tuning rather than model blame. Practitioners should calibrate judges before using them for release decisions.

Agentic AI inherits the same trust problem seen in overprivileged service automation. When the system can act externally, the real risk is not only what it says but what it can trigger. That makes alignment between intent, tool scope, and parameter validity essential. The field needs more than prompt engineering: it needs a lifecycle view of agent permissions, testing, and revocation. Practitioners should govern agents as scoped actors with measurable boundaries.

Tool-use evaluation will converge with identity governance as agents move into production. The article’s workflow shows a practical path from experiment to policy enforcement, but the underlying direction is broader: once agents interact with business systems, their actions need the same scrutiny applied to privileged automation. That links agent evaluation to NHI governance, workload identity, and least privilege. Practitioners should prepare for agent controls to sit alongside IAM and PAM rather than outside them.

What this signals

Separate measurement is the control that will distinguish mature agentic programmes from experimental ones. As tool-calling becomes operational, teams that only track a single success score will miss whether failures are happening at selection, invocation, or rubric calibration. That is why evaluation design now belongs in governance discussions, not just model testing workflows.

The next phase of agentic AI governance will look less like prompt tuning and more like scoped entitlement management. Once tools connect to business systems, the evaluation stack has to verify permission boundaries, argument integrity, and revocation paths. Teams should align this work with the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10.

Agent evaluation will increasingly overlap with NHI governance: if an agent can act externally, it needs the same lifecycle discipline as any other privileged non-human actor. That means scoping, testing, and offboarding are not optional extras. Practitioners should prepare to govern agents as access-bearing systems with measurable boundaries.

For practitioners

• Implement separate tool-selection and tool-invocation scores Track whether the model picked the correct tool and whether it passed valid arguments as different control outcomes. Do not collapse them into one metric, because a single aggregate score hides where the workflow is failing.
• Calibrate judges against semantic equivalence Review cases where different values are still operationally valid, such as location names or equivalent entity names. Tune the evaluator so it reflects business intent instead of strict string matching.
• Add domain-specific constraints to evaluator prompts Encode rules that matter in production, such as current-year date handling, required parameter formats, and disallowed hallucinated values. Re-run the benchmark after each change to see whether the evaluator is catching the right failures.
• Treat agents as governed non-human identities Assign explicit tool scope, review boundaries, and revocation criteria for any agent that can trigger external actions. Use the same discipline you would apply to high-risk machine identities.

Key takeaways

• Tool-calling agents fail in two distinct ways, so one aggregate score is not enough to tell you where the risk sits.
• Reference-free evaluators are useful, but they must be calibrated against real operational intent or they will misclassify valid behaviour.
• As agents gain tool access, evaluation becomes part of identity and access governance for non-human actors.

Key terms

• Tool Selection Accuracy: Tool selection accuracy measures whether an agent chooses the right tool for a given task, not merely whether the tool responds successfully. It is a governance signal because repeated wrong-tool choices can create cost, security, and compliance problems even when the end output appears acceptable.
• Tool Invocation: Tool invocation is an action where an AI agent calls an external system such as a database, API, or file service. Each invocation should be treated as an auditable identity action because it is the point where the agent can move data, trigger changes, or widen its reach across the environment.
• Reference-Free Evaluation: A scoring approach that judges output without a gold standard answer. It is used for open-ended tasks where the right response is not fixed, including chatbots and agentic workflows. The trade-off is that scoring quality depends on the rubric, the judge model, and the coverage of real-world scenarios.
• Semantic Equivalence: Semantic equivalence means two different values or phrases can still be operationally the same for the task at hand. In evaluation, this matters because strict string matching can mark a correct tool call as wrong when the model uses a valid alternative such as a recognised location name or entity alias.

What's in the full article

Arize's full tutorial covers the operational detail this post intentionally leaves for the source:

• The exact Phoenix evaluator templates for tool selection and tool invocation, including how the judge input is structured.
• The notebook workflow for loading the travel assistant dataset, running experiments, and comparing versions in the UI.
• The prompt edits used to fix year handling and calibrate semantic equivalence for location arguments.
• The custom evaluator pattern for matching tool calls against labeled ground truth without relying on string equality.

👉 Arize's full walkthrough includes the Phoenix evaluator setup, iteration examples, and the notebook used for the demo.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It helps practitioners apply identity controls to non-human systems that can act in production.