How should teams evaluate MCP tool discovery in production?

Measure retrieval accuracy, selection accuracy, latency, and token use together. A discovery method that saves tokens but frequently hides the correct tool may look efficient while still degrading reliability. The right evaluation asks whether the agent can consistently surface and choose the intended capability under real catalog size.

How to evaluate discovery quality in a real MCP catalog

Production evaluation should treat discovery as a retrieval and ranking problem, not just a token-efficiency problem. Teams need to test whether the agent can surface the right tool, in the right context, often enough that downstream task success stays stable as the catalog grows. That means using representative queries, realistic tool descriptions, and the same discovery path the agent will use in production.

The practical unit of measurement is the end-to-end outcome: can the agent find the intended capability, choose it, and proceed without unnecessary detours? Retrieval accuracy and selection accuracy capture different failure modes, and they should be reported separately rather than blended into a single convenience metric. A method that improves compression but obscures the intended tool is not a win.

Catalog scale matters because discovery can look strong in a small lab while degrading under production entropy. As the number of tools, tool variants, and near-duplicates increases, the evaluation should check whether ranking still keeps the intended tool visible near the top, whether tie-breaking remains stable, and whether the agent over-selects generic but wrong tools when the catalog becomes crowded.

What failure modes matter most in production

Discovery breaks in a few repeatable ways: the correct tool is omitted, the agent retrieves too many plausible tools, a generic tool displaces a more specific one, or latency becomes high enough that the agent times out, retries, or truncates context. The most useful tests simulate these conditions explicitly instead of only measuring average-case performance.

Tool descriptions and metadata are part of the security and reliability surface because they steer ranking. If naming is vague, overlapping, or inconsistent, the agent may appear to be “choosing intelligently” while actually being nudged by noisy labels. That is why evaluation should include tools with similar names, partial overlaps, and ambiguous scope, not just cleanly separated capabilities.

Production teams should also watch for hidden brittleness in fallback behavior. If discovery silently falls back to a broad catch-all tool, the system may preserve task completion while weakening precision, governance, and observability. That can be acceptable for some workflows, but only if the fallback is intentional and measured, not accidental.

How to make the evaluation operationally useful

Use a test set that reflects the actual production catalog, including new tools, deprecated tools, and tools with overlapping function. Then score each query across retrieval accuracy, selection accuracy, latency, and token use together so the team can see the trade-off surface rather than optimizing one number at the expense of the others. If a discovery strategy changes ranking quality, measure its effect on the downstream task, not just on the retrieval step.

It also helps to track whether performance changes across slices such as tool count, query type, and tool similarity. A discovery method that works for unique tools may fail when two tools provide nearly the same capability. In that case, the evaluation should show whether the system can still prefer the intended tool, or whether it drifts toward whichever option is cheaper to surface.

Where production risk is high, teams should keep a manual review path for edge cases and regression testing for catalog changes. Discovery quality is not a one-time benchmark; it should be rechecked whenever tool definitions, namespaces, permissions, or routing logic change. The MCP Security Guide is useful here because discovery quality and authorization behavior often fail together when tool metadata and access policy drift out of sync.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Agentic AI Top 10 and OWASP API Security Top 10 address the attack and risk surface, while NIST CSF 2.0 sets the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Agentic AI Top 10	ASI02 — Tool Misuse	MCP discovery quality affects whether agents pick the right tool.
	ASI03 — Identity & Privilege Abuse	Tool selection in MCP can change what authority an agent exercises.
	ASI10 — Rogue Agents	Poor discovery can let agents act on unintended tools or capabilities.
Recommendation — Test discovery paths to reduce wrong-tool selection and unsafe tool invocation. Validate that discovered tools do not expand agent privilege beyond intent. Constrain discovery so agents cannot autonomously pivot to unintended actions.
OWASP API Security Top 10	API9 — Improper Inventory Management	MCP discovery depends on accurate tool inventory and visibility.
Recommendation — Maintain an accurate tool inventory and retire stale entries from discovery.
NIST CSF 2.0	ID.AM-01 — Physical devices and systems within the organization are inventoried	Discovery evaluation depends on knowing what tools exist in the catalog.
Recommendation — Inventory all MCP tools and keep the catalog current before evaluating discovery.

Practitioner Guidance

What to prioritize: Judge discovery by whether the intended tool stays findable under realistic catalog size, not by whether the prompt path looks elegant in a demo. The fastest scheme is not the best one if it pushes the correct capability out of the candidate set.

What to verify: Check that your evaluation harness measures both retrieval and selection, and that the same benchmark reports latency and token consumption. Those four signals together reveal whether a discovery change is truly improving usability or simply shifting cost around.

Common mistake: Teams often optimize for fewer tokens or shorter lists and accidentally reduce the probability that the right tool is even considered. That trade-off is easy to miss unless the benchmark includes confusing, overlapping, and near-duplicate tools.

What good looks like: The agent consistently surfaces the intended tool near the top, selects it when appropriate, and does so with stable latency as the catalog grows. If the answer depends on a hidden fallback or a narrow lab catalog, the discovery method is not production-ready.

Practitioner takeaway: Treat MCP discovery as a reliability control, not a compression exercise, and keep the benchmark tied to real catalog entropy so efficiency never masks missed capability.