MCP tool search accuracy gaps are reshaping agent runtime design

At a glance

What this is: Stacklok compares two MCP tool-discovery approaches and finds that hybrid search materially improves correct tool selection at scale.

Why it matters: For IAM, NHI, and agentic AI teams, this matters because tool discovery is now part of runtime privilege control, not just an optimisation problem.

By the numbers:

• Stacklok’s testing across 2,792 MCP tools found 94% accuracy in selecting the right tools with MCP Optimizer, compared with 34% for Anthropic’s Tool Search Tool.
• MCP Optimizer reduced average input tokens to 3,296 per request, versus 2,823 for Anthropic’s BM25 variant and 13.55 seconds for regex-based search.

👉 Read Stacklok's comparison of MCP Optimizer and Anthropic's Tool Search Tool

Context

MCP tool discovery is the control layer that decides which capabilities an AI agent can see and use during a session. In this article, the core problem is not raw model intelligence but tool sprawl, where loading hundreds or thousands of tool definitions wastes tokens and makes correct selection less reliable for production AI agent workflows.

For identity and access teams, that shifts the discussion from model quality to runtime authorisation shape. When an agent only loads a subset of tools on demand, the governance question becomes whether discovery is accurate, constrained, and auditable enough to support least privilege for autonomous or semi-autonomous execution.

The article compares two ways of narrowing the tool set, one hybrid semantic-plus-keyword approach and one search-only approach, then measures how often each finds the correct tool. That is a typical challenge for teams trying to operationalise MCP at scale, not an edge case.

Key questions

Q: How should security teams handle tool discovery for AI agents in MCP environments?

A: Security teams should treat tool discovery as a privilege boundary, not a convenience layer. Limit the tools an agent can see, defer loading until a task requires it, and log which capabilities were exposed before execution. That reduces context overload and makes misselection easier to detect and investigate.

Q: Why do large MCP tool catalogs increase risk for AI agents?

A: Large catalogs increase risk because they widen the tool surface an agent can discover, while also making correct retrieval harder. The result is more context pressure, more failed searches, and a greater chance that the agent sees or uses the wrong capability. In practice, scale changes tool discovery into an identity governance issue.

Q: What do teams get wrong about token reduction in agent workflows?

A: Many teams assume lower token usage automatically means safer or better governed agent behaviour. It does not. Token reduction only helps if the system still surfaces the right tool and keeps unnecessary capabilities out of scope. Efficiency should be measured alongside retrieval precision, auditability, and the operational impact of search failures.

Q: How do I know whether MCP search is actually working well enough?

A: You know MCP search is working when the correct tool appears consistently for real user intents, missed retrievals are rare, and the loaded tool set stays appropriately small. If you need repeated retries, see unexpected tool choices, or cannot audit what was loaded, the search layer is not yet reliable enough for production.

Technical breakdown

Why MCP tool discovery becomes a privilege boundary

In MCP, tool definitions are not just metadata. They are the executable capability catalogue that an agent can draw from during a session. When all tools are loaded, the agent sees a broad privilege surface, which inflates context and increases the chance of confusion. When tools are searched and loaded on demand, the search step becomes part of the authorisation path: if discovery fails, the agent may not access the right tool, and if discovery is too broad, the agent sees more capability than necessary. That makes tool search a governance control, not a convenience feature.

Practical implication: treat MCP discovery as a policy-bearing control point and review which tools an agent can see before it can act.

Hybrid semantic and keyword search versus narrow retrieval

The article contrasts a hybrid semantic-plus-BM25 retrieval method with search methods that rely on BM25 alone or regex patterns. Hybrid retrieval matters because tool names, descriptions, and user intent rarely match exactly. Semantic search helps with meaning, while keyword ranking preserves precision on exact tool names and functions. The article’s results show that retrieval quality drives selection quality: if the correct tool is not surfaced, the model cannot choose it, no matter how capable the model is. That makes search design central to reliable MCP operation.

Practical implication: evaluate retrieval quality on real tool corpora, not only model benchmarks, before putting MCP search into production.

Why token savings do not equal operational safety

Reducing input tokens helps with cost and latency, but it does not solve the underlying governance issue. A smaller context window can still expose the wrong tool, and a larger catalog can still hide the right one. The article shows that a system can improve efficiency while retaining a meaningful failure mode in tool retrieval. For identity teams, that is the key lesson: optimisation and control are different outcomes. Tool loading must be measured for both efficiency and correct capability selection, especially in environments where a wrong tool could trigger unintended data access or actions.

Practical implication: pair token-efficiency metrics with retrieval-accuracy tests and audit the consequences of false positives and false negatives.

NHI Mgmt Group analysis

MCP tool discovery is now a runtime access-control problem, not a simple search problem. The article shows that agents do not just need fewer tokens, they need the right tools surfaced at the right time. That moves MCP governance into the same category as capability scoping and access review, because tool visibility determines what the agent can attempt. Teams should treat search quality as part of the non-human identity control plane.

Tool retrieval accuracy is the real control variable in large MCP estates. A 94% selection rate versus 34% changes the operational risk profile entirely. When the wrong tool is hidden or the right tool is not surfaced, the model cannot compensate with reasoning alone. This is the same structural issue seen in other NHI patterns: if discovery fails, downstream control design becomes irrelevant.

Token efficiency without selection fidelity creates a false sense of readiness. The article correctly frames context-window pressure as a production constraint, but the deeper issue is whether the agent can make bounded choices from a governed tool set. That is where least privilege becomes measurable in agent systems. Practitioners should interpret efficiency gains as useful only when paired with demonstrable retrieval precision.

Named concept: tool discovery blast radius. The more tools an agent can discover, the larger the potential misuse surface if search ranks the wrong capability or surfaces too many choices. This is not just context bloat, it is an access-surface expansion problem. Teams need to think about blast radius in terms of tool visibility, not only tool execution.

For agentic AI programmes, MCP search quality will increasingly sit alongside identity governance, not inside the model stack. The article shows that retrieval architecture determines runtime behaviour. That means the security conversation must include tool catalog design, search controls, and auditability of loaded capability sets. Practitioners should align MCP governance with NHI control thinking before deployments scale further.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface.
• Another 33% of organisations report that AI agents have accessed inappropriate or sensitive data beyond their intended scope, reinforcing that runtime scope drift is already observable in deployed environments.
• That is why OWASP Agentic AI Top 10 is a useful forward reference when teams are deciding how to bound tool use and agent behaviour.

What this signals

Tool discovery blast radius: the real design variable is not how many tools an agent can theoretically use, but how much capability it can discover in one runtime path. As more teams connect agents to MCP servers, the governance model needs to shift toward scoped discovery, auditable loads, and retrieval testing that mirrors production intent. The search layer is becoming part of the identity perimeter, not an implementation detail.

With 80% of organisations already reporting AI agents acting beyond intended scope in the SailPoint study, the next control question is whether discovery and selection are bounded tightly enough to prevent accidental capability expansion. The practical programme response is to measure agent sessions for wrong-tool surfacing, not just token spend, and to align that work with NIST AI Risk Management Framework governance expectations.

Teams that are already tracking NHI lifecycle and secrets governance should extend the same discipline to MCP tool catalogs. If a tool list cannot be reviewed, versioned, and tested like any other privileged access surface, it will eventually behave like one.

For practitioners

• Measure tool retrieval before scaling agent workloads Build test sets from your actual MCP servers and verify whether the correct tool appears in search results under realistic prompts. Track retrieval accuracy separately from selection accuracy so you can see whether failures come from discovery or model choice.
• Constrain tool visibility by task scope Do not expose entire tool catalogs to every session. Group tools by function, environment, or data domain so the discovery layer only searches within the smallest useful set of capabilities.
• Audit false positives and missed tools Review cases where the wrong tool is surfaced or the right tool is absent, then classify the operational impact. A missed retrieval event in an AI agent is a governance failure, not only a performance issue.
• Pair cost metrics with control metrics Use token counts and runtime only as secondary indicators. Your primary measures should be correct tool surfacing, auditable tool loads, and whether a search failure could expose unauthorised capability.

Key takeaways

• MCP tool search is an access-control problem in disguise because discovery determines what an agent can do at runtime.
• Stacklok’s results show that retrieval quality, not model size alone, drives whether tool selection remains reliable at scale.
• Enterprises should govern MCP tool catalogs with the same discipline they use for NHI privilege boundaries and auditability.

Key terms

• MCP Tool Discovery: The process by which an MCP server or proxy exposes available tools to a caller. If discovery is not filtered by principal and context, it can reveal capabilities that the identity is not actually authorised to use, creating an exposure before execution begins.
• Tool Retrieval Accuracy: The rate at which a search mechanism surfaces the correct tool for a given task. For AI agents, this is a control quality measure, not just a model metric, because poor retrieval changes what the agent can do and can force retries, mis-selections, or failed workflows.
• Context window bloat: The accumulation of intermediate data, prompts, and outputs that an AI model must carry while completing a task. As bloat grows, reliability falls and the agent becomes harder to govern because more of the work happens in transient model memory rather than in the source system.
• Tool Discovery Blast Radius: The amount of unintended capability an agent can expose to itself when a search layer surfaces too broad a tool set. The larger the blast radius, the greater the chance that a wrong or over-privileged tool is chosen, especially in large MCP estates.

What's in the full article

Stacklok's full article covers the operational detail this post intentionally leaves for the source:

• Full benchmark tables comparing retrieval and selection accuracy across tool-retrieval methods
• The test methodology used across 2,792 MCP tools, including synthetic query generation
• Runtime performance measurements that break down token usage, average tools retrieved, and execution time
• Implementation context for ToolHive and MCP Optimizer that matters to teams evaluating deployment options

👉 Stacklok's full post covers the benchmark results, runtime trade-offs, and implementation context.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.