How should teams evaluate MCP tool discovery in production?

How to evaluate discovery quality in a real MCP catalog

Production evaluation should treat discovery as a retrieval and ranking problem, not just a token-efficiency problem. Teams need to test whether the agent can surface the right tool, in the right context, often enough that downstream task success stays stable as the catalog grows. That means using representative queries, realistic tool descriptions, and the same discovery path the agent will use in production.

The practical unit of measurement is the end-to-end outcome: can the agent find the intended capability, choose it, and proceed without unnecessary detours? Retrieval accuracy and selection accuracy capture different failure modes, and they should be reported separately rather than blended into a single convenience metric. A method that improves compression but obscures the intended tool is not a win.

Catalog scale matters because discovery can look strong in a small lab while degrading under production entropy. As the number of tools, tool variants, and near-duplicates increases, the evaluation should check whether ranking still keeps the intended tool visible near the top, whether tie-breaking remains stable, and whether the agent over-selects generic but wrong tools when the catalog becomes crowded.

What failure modes matter most in production

Discovery breaks in a few repeatable ways: the correct tool is omitted, the agent retrieves too many plausible tools, a generic tool displaces a more specific one, or latency becomes high enough that the agent times out, retries, or truncates context. The most useful tests simulate these conditions explicitly instead of only measuring average-case performance.

Tool descriptions and metadata are part of the security and reliability surface because they steer ranking. If naming is vague, overlapping, or inconsistent, the agent may appear to be “choosing intelligently” while actually being nudged by noisy labels. That is why evaluation should include tools with similar names, partial overlaps, and ambiguous scope, not just cleanly separated capabilities.

Production teams should also watch for hidden brittleness in fallback behavior. If discovery silently falls back to a broad catch-all tool, the system may preserve task completion while weakening precision, governance, and observability. That can be acceptable for some workflows, but only if the fallback is intentional and measured, not accidental.

How to make the evaluation operationally useful

Use a test set that reflects the actual production catalog, including new tools, deprecated tools, and tools with overlapping function. Then score each query across retrieval accuracy, selection accuracy, latency, and token use together so the team can see the trade-off surface rather than optimizing one number at the expense of the others. If a discovery strategy changes ranking quality, measure its effect on the downstream task, not just on the retrieval step.

It also helps to track whether performance changes across slices such as tool count, query type, and tool similarity. A discovery method that works for unique tools may fail when two tools provide nearly the same capability. In that case, the evaluation should show whether the system can still prefer the intended tool, or whether it drifts toward whichever option is cheaper to surface.

Where production risk is high, teams should keep a manual review path for edge cases and regression testing for catalog changes. Discovery quality is not a one-time benchmark; it should be rechecked whenever tool definitions, namespaces, permissions, or routing logic change. The MCP Security Guide is useful here because discovery quality and authorization behavior often fail together when tool metadata and access policy drift out of sync.

Practitioner Guidance

What to prioritize: Judge discovery by whether the intended tool stays findable under realistic catalog size, not by whether the prompt path looks elegant in a demo. The fastest scheme is not the best one if it pushes the correct capability out of the candidate set.

What to verify: Check that your evaluation harness measures both retrieval and selection, and that the same benchmark reports latency and token consumption. Those four signals together reveal whether a discovery change is truly improving usability or simply shifting cost around.

Common mistake: Teams often optimize for fewer tokens or shorter lists and accidentally reduce the probability that the right tool is even considered. That trade-off is easy to miss unless the benchmark includes confusing, overlapping, and near-duplicate tools.

What good looks like: The agent consistently surfaces the intended tool near the top, selects it when appropriate, and does so with stable latency as the catalog grows. If the answer depends on a hidden fallback or a narrow lab catalog, the discovery method is not production-ready.

Practitioner takeaway: Treat MCP discovery as a reliability control, not a compression exercise, and keep the benchmark tied to real catalog entropy so efficiency never masks missed capability.

Related resources from NHI Mgmt Group

• How should security teams govern AI agent tool access when MCP uses a loosely typed transport?
• Should security teams treat MCP tool calls like ordinary API requests?
• How should teams decide whether to centralise model access and MCP tool calls through one gateway?
• How should HIPAA-oriented teams evaluate DLP coverage for copilots and MCP tools?