MCP vs CLI skills for agents: what evaluation results show

At a glance

What this is: Arize’s evaluation found that MCP, skills, and CLI access can deliver similar task accuracy, but they differ sharply in cost, latency, and composability.

Why it matters: IAM, PAM, and NHI teams need to understand that tool access is not just an engineering choice, because protocol shape, auth model, and runtime flexibility all affect control boundaries and operational risk.

By the numbers:

• Only 18% of MCP server deployments implement any form of access scoping for tool permissions.
• 53% of MCP servers expose credentials through hard-coded values in configuration files.

👉 Read Arize’s evaluation of MCP vs CLI skills for agent tool access

Context

MCP, CLI skills, and shell access are all ways for an agent to reach tools, but they do not create the same identity and governance problem. The core issue is not whether the agent can complete the task. It is which control model governs tool discovery, authorization, auditability, and revocation when the agent acts at runtime.

For IAM and NHI programmes, this is a practical boundary question. A remote protocol such as MCP tends to centralise access and consent, while CLI-based access often shifts control into local tokens, scripts, and environment state. That difference matters because the same agent may look equivalent in a benchmark yet create very different governance overhead in production.

The article’s evaluation is useful precisely because it moved the debate out of opinion and into measured trade-offs. Its starting point is typical for teams building AI-enabled workflows: accuracy is only one part of the control picture, and the more important questions are whether access is scoped, observable, and reversible.

Key questions

Q: What breaks when an agent has to fall back from MCP to shell access?

A: The main failure is control drift. Once the agent leaves the bounded protocol and starts composing shell commands, the identity path becomes harder to scope, harder to audit, and easier to overuse. That is not automatically unsafe, but it means the governance model must cover the fallback path, not just the primary tool.

Q: Why do remote protocols and CLI access create different IAM risks for agents?

A: Remote protocols can centralise consent and access scope, while CLI access often depends on local tokens, scripts, and developer-managed configuration. The risk changes because the credential location, revocation path, and audit trail change. Teams should evaluate the whole access pathway, not just the tool name.

Q: How can security teams tell whether an agent tool surface is too narrow?

A: Look for repeated tool fan-out, large JSON payloads, and frequent fallback into broader execution paths. Those signals show the agent is compensating for a non-composable surface. When that happens, the issue is not accuracy alone. It is that the identity model is no longer aligned with the task shape.

Q: How should organisations decide between CLI and MCP for agent automation?

A: Use CLI when the agent is acting as a single developer and the credentials are intentionally that person’s own. Use MCP when the workflow is shared, customer-facing, or production-bound, because the ability to scope, log, and revoke access becomes more important than raw token efficiency.

Technical breakdown

Why fixed tool surfaces change agent behaviour

A fixed tool surface such as an MCP server exposes a bounded set of calls. That works well when the task maps cleanly to those calls, but it becomes inefficient when the agent must combine several steps that the protocol does not compose natively. In those cases, the agent fans out into many calls, often returning verbose JSON and reprocessing context at each turn. The result is more token use, more latency, and a larger operational trace. By contrast, a shell can compose commands with pipes, filters, and local parsing, so the agent can build the missing logic itself.

Practical implication: map tool surfaces to the kinds of tasks they can actually compose, not just to the tasks they can expose.

How authentication models shape governance

CLI access and MCP access sit behind very different auth assumptions. CLI workflows often depend on local tokens, manual configuration, and environment-specific credential handling. MCP, by contrast, is designed to work with OAuth-style consent and centralised access control, which makes it easier to scope what a tool can reach and to revoke access later. That difference is not cosmetic. It determines whether an agent’s access is a local convenience or a governed enterprise entitlement. For identity teams, the important distinction is whether the access path can be inspected and constrained as part of the broader IAM model.

Practical implication: treat agent tool access as an identity pathway and classify it by how consent, scope, and revocation are enforced.

Why auditability becomes the deciding factor at scale

When an agent relies on shells, scripts, and multiple chained calls, the operational trace spreads across commands, files, and intermediate outputs. That can still be auditable, but the evidence is fragmented unless the programme deliberately aggregates it. A protocol-driven path can make audit easier if it is well instrumented, but it can also conceal gaps if the agent silently falls back to a different tool. In both cases, governance breaks when teams assume the tool boundary is the control boundary. The real control boundary is the combination of identity, permission, and traceability across every execution path.

Practical implication: verify that logs, approvals, and revocation coverage follow the agent across every fallback path, not only the primary tool.

Threat narrative

Attacker objective: The objective is to obtain broader, harder-to-govern tool access that increases the agent’s ability to act across systems without clear control boundaries.

1. Entry occurs when an agent is granted access to a remote protocol, CLI, or shell environment and begins using it as a tool interface. Escalation happens when the agent compensates for missing composition by chaining more calls, parsing outputs locally, or switching to broader shell capabilities. Impact follows when that broader access path expands the attack surface, increases credential exposure, or weakens auditable control over what the agent can do.

Breaches seen in the wild

• Meta AI Instagram Account Takeover — 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

MCP versus CLI is not a tooling preference, it is an identity governance decision. The article shows that both approaches can reach similar correctness, but they do so under different control assumptions. CLI access often shifts privilege into local tokens and command composition, while MCP centralises tool exposure but can still be bypassed when the task is not composable. Practitioners should read this as a governance design choice, not an engineering taste debate.

Tool composability has become a new form of identity blast radius. When an agent cannot complete a task within a bounded API, it expands into more calls, more context, and often more powerful fallback tools. That expansion changes the audit surface and the permission surface at the same time. The practical conclusion is that the blast radius of an agent is determined less by the model and more by how much runtime composition the surrounding control plane permits.

Access scoping for agents is still too often treated as a protocol feature instead of a lifecycle control. An MCP server with OAuth-style consent can be better governed than a tokenised CLI, but only if provisioning, revocation, and review are actually tied to the agent’s runtime access path. This is where NHI governance meets IAM operations: the question is not which tool exists, but whether its identity can be offboarded as cleanly as it was onboarded.

Opinionated skills and remote protocols will coexist because they solve different governance problems. The article’s strongest conclusion is that the market should stop treating one access pattern as universally superior. Local CLI execution supports composition and developer workflows, while remote protocols support central control and customer-facing delegation. Teams should therefore classify agent access by who controls the tool, where the credential lives, and how quickly the privilege can be withdrawn.

Named concept: runtime tool surface drift. The more an agent must escape its initial tool boundary to finish a task, the more its effective privilege set changes during execution. That drift is the governance issue hiding inside many agent evaluations, and it is why access review alone cannot explain operational risk. Practitioners should treat runtime drift as a control problem, not a performance quirk.

From our research:

• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to AI Agents: The New Attack Surface report.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• That gap is why the OWASP Agentic Applications Top 10 matters for runtime access, tool misuse, and identity boundary design.

What this signals

With 80% of organisations already seeing AI agents act beyond intended scope in the SailPoint research, the governance problem is no longer about future adoption. It is about whether current IAM, PAM, and NHI controls can observe, constrain, and revoke runtime access before agents start behaving like unreviewed privileged operators.

Runtime tool surface drift: when an agent must escape its primary tool boundary to finish a task, the effective privilege set changes mid-execution. That means control design has to follow the execution path, not just the declared tool list, and identity teams should expect audit models to lag unless every fallback is instrumented.

Enterprises should expect more hybrid tool patterns, not fewer. The practical response is to align protocol choice, credential storage, and revocation mechanics with the task class, then document where the agent may legally change execution mode so governance can keep pace.

For practitioners

• Map agent tool paths to identity controls Inventory which workflows use MCP, which use CLI, and which silently fall back to shell execution. Tie each path to a named owner, a credential source, and a revocation process so the access model is explicit before deployment.
• Constrain fallback paths before broad rollout If an agent can switch from a protocol tool to bash or local parsing, treat that as a privileged path that needs approval, logging, and containment. The fallback path is often where audit boundaries disappear.
• Review token handling and secret exposure points Check where local tokens, OAuth consent, and environment variables are stored, rotated, and revoked. The article’s comparison shows that the control model changes when credentials move from central consent to developer-managed configuration.
• Measure composition cost as a governance signal Track tool-call count, context growth, and timeout frequency alongside correctness. A sharp rise in those signals often means the agent has exceeded the intended scope of the tool surface and is compensating through broader access.

Key takeaways

• Agent tool choice is an identity problem because it changes who controls access, where credentials live, and how easily privilege can be withdrawn.
• Benchmarked correctness can hide major control differences, especially when an agent quietly falls back from a bounded protocol to a broader shell path.
• Teams should govern tool surfaces as part of NHI lifecycle management, with explicit scope, logging, and revocation for every execution path.

Key terms

• Assistant Tool Surface: The set of files, commands, APIs, renderers, and outbound paths an AI assistant can reach during a session. Security depends on how these tools are segmented, approved, and logged, because the danger often comes from combining ordinary capabilities into an abusive workflow.
• Runtime Tool Surface Drift: A change in the effective privilege boundary that happens while an agent is executing, usually because it leaves the primary tool path and starts composing with broader commands or fallback tools. It is a governance issue because the controlled path no longer matches the actual path.
• Fallback Path: A secondary access route used when the primary authentication method fails. Fallback paths matter because they often become the real control in day-to-day use. If they are easier than the intended method, the organisation will drift toward them and weaken its identity posture.
• Tool Fidelity: The degree to which an agent stays within the tools it was intended to use. High tool fidelity is useful, but only when the allowed tools themselves are properly scoped and the agent is not escaping into unmanaged execution paths.

What's in the full article

Arize’s full article covers the operational detail this post intentionally leaves for the source:

• The full evaluation design, including the 25-task GitHub benchmark and the four tool-access arms used in testing.
• The detailed cost and latency comparisons across task tiers, including where MCP fan-out created higher token and wall-clock overhead.
• The per-task examples showing when MCP was more efficient and when CLI or baseline command execution performed better.
• The authors’ reasoning on when protocol-based access makes more sense than shell-based access for real deployments.

👉 Arize’s full article covers the benchmark setup, tier-by-tier results, and the cost trade-offs behind each tool pattern.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.