Computer-Use Agents vs API-Integrated Agents: Choosing the Safer Interface for Each Task

An AI agent that drives a screen and an agent that calls an API can complete the same business task, but they expose completely different credentials, read completely different untrusted input and leave completely different audit trails. Computer-use agents see pixels, move a pointer and type into whatever application is in front of them, usually inside a signed-in browser session. API-integrated agents call defined tools or endpoints with tokens that can be scoped to one resource and a few operations. This guide compares the two interfaces on credential exposure, injection surface, least privilege, sandboxing and auditability, gives a worked scenario and a decision checklist, and explains when a screen-driving agent is still the right choice. It is for architects, AI platform teams and security reviewers approving agent designs. For the controls specific to browser and desktop agents, see the Browser and Computer-Use Agent Guide; for per-call enforcement on API tools, see Tool-Call Security for AI Agents.

Key takeaways

• Prefer the API when one exists. A scoped, audience-bound token on a defined tool gives you least privilege, structured logs and a policy enforcement point. A screen gives the agent everything the signed-in user can see and do.
• Session cookies are the computer-use credential. A computer-use agent usually works inside an authenticated session, so anything that steers it, or steals from its environment, inherits the full user session rather than one scope.
• Every pixel is untrusted input. Web pages, documents, emails and images on screen can carry instructions. The injection surface of a computer-use agent is the whole visible environment, not just tool results.
• Sandboxing is mandatory, not optional, for screens. All three major model providers' computer-use documentation tells developers to run these agents in an isolated virtual machine, container or dedicated browser profile, with limited network access and human confirmation for consequential actions.
• Use computer use as a bridge. It is valuable for legacy applications with no API, and for short-lived automation while an integration is built. Treat each computer-use workflow as technical debt with an owner and an exit date.

The short answer

Use an API-integrated agent whenever the target system offers an API, an MCP server or another defined integration, because only then can you give the agent a credential limited to specific resources and operations, enforce policy on each call and log structured actions. Use a computer-use agent only where no API exists or the task is genuinely visual (checking how a page renders, operating a legacy desktop application), and then run it in an isolated environment, with a dedicated low-privilege account, a domain allowlist and human approval for anything that moves money, sends data or changes records. If a design proposes computer use against a system that has an API, ask why; the answer is often convenience, not necessity.

Comparison table

Credential exposure: session cookies vs scoped tokens

A computer-use agent normally works where a person would: inside a browser that is signed in. That session carries every permission the account has in every application it can reach through single sign-on, and the cookies that represent it are bearer credentials. ATT&CK tracks the related techniques as T1539 Steal Web Session Cookie and T1185 Browser Session Hijacking. If the agent's environment is the user's own desktop profile, it also inherits saved passwords and other signed-in tabs. Anthropic's computer-use documentation advises avoiding giving the model access to sensitive data such as account login information; OpenAI's guidance says to keep credentials out of page text and screenshots. See Session Cookies vs Bearer Tokens and the Session Hijacking and Cookie Theft Guide.

An API-integrated agent can hold a token that names one resource, carries a small set of scopes and expires quickly. The MCP authorization specification (version 2025-11-25) requires clients to send the RFC 8707 resource parameter identifying the target server, requires servers to accept only tokens issued for them and forbids passing a received token through to upstream APIs. RFC 9700, the OAuth 2.0 security best current practice, says access tokens should be audience-restricted and their privileges restricted to the minimum needed, and recommends sender-constraining tokens with mutual TLS or DPoP. None of these controls has a direct equivalent for a cookie-based screen session. See the MCP Authorisation Specification Guide and Secrets in Prompts vs Brokered Credentials.

The practical difference is blast radius. A stolen or misused agent token for a ticketing API lets an attacker read and update tickets until it expires. A misused computer-use session lets them do whatever the account can do across every application it is signed into.

Injection surface

Both agents are exposed to prompt injection (OWASP LLM01:2025 Prompt Injection; ATT&CK T1698 Indirect Prompt Injection; ATLAS AML.T0051.001 Indirect), but the surface differs in size and shape.

• Computer-use: every web page, document, email, pop-up and image rendered on screen is input. Anthropic's documentation warns that instructions on web pages or contained in images might override the developer's instructions. Hidden or low-contrast text and instructions embedded in images are read by the model even when a human reviewer would miss them. See the Multimodal Prompt Injection Guide.
• API-integrated: the surface is the content returned by tools and retrieval: ticket text, email bodies, file contents. It is still untrusted, but it arrives as data in a known field, so you can label it, filter it and stop it from reaching tools that act on its say-so.

The more important difference is what an injection can achieve. With an API agent, a gateway can refuse a call that is outside policy whatever the model was persuaded to do. With a computer-use agent, the action space is the whole user interface, so a successful injection can reach any button the account can click. That is OWASP LLM06:2025 Excessive Agency in its clearest form. See the OWASP LLM Top 10 LLM06 Excessive Agency Guide.

Threat model and failure modes for each option

Computer-use agent

• Steering through content: a page or document tells the agent to visit a URL, download a file or submit a form, and the agent complies.
• Data exfiltration by typing: the agent copies data into a web form, chat box or search field on an attacker-controlled site. OpenAI's guidance treats typing sensitive information into a form as data transmission that needs user confirmation.
• Session theft from the environment: if the sandbox is the user's real profile, malware or a malicious extension in that profile reaches the session too.
• Wrong-target actions: layout changes or similar-looking elements lead to clicks on the wrong record, with no error returned.
• Consent and terms: the agent accepts cookies, terms of service or a purchase on the user's behalf.

API-integrated agent

• Argument manipulation: injected content changes tool parameters, for example a recipient address or an amount.
• Over-scoped tokens: a single broad token, or a static key copied into configuration, turns a narrow integration into a wide one.
• Token passthrough and confused deputy: an intermediary forwards the user's token to an upstream API, which the MCP specification forbids.
• Tool sprawl: the agent is connected to many tools it rarely needs, each widening what a successful injection can reach.

Least privilege feasibility

Least privilege (NIST SP 800-53 AC-6) depends on having something smaller than the whole account to grant. API integrations offer it: scopes, resource indicators, role assignments and per-call policy. A tool can be read-only, limited to one project and expire in minutes. See AI Agent Service Accounts vs On-Behalf-Of Access.

Computer use can only approximate least privilege by building a smaller world around the agent:

• Dedicated account: a separate agent account with the minimum application roles, never a person's own account or an administrator.
• Domain allowlist: network rules that let the environment reach only the sites the task needs. Anthropic and OpenAI both recommend an allowlist of domains or sites.
• Action restriction: restricting the operations the harness exposes, for example screenshots and clicks but no typing, where the provider's toolkit allows it.
• Short-lived sessions: a fresh environment per task, with sessions closed and released when the run ends, as OpenAI's guidance recommends.

Sandboxing and provider safeguards

All three major model providers publish computer-use safety guidance that converges on the same pattern. Providers are listed alphabetically; the order is not a ranking.

• Anthropic: the computer use tool documentation recommends a dedicated virtual machine or container with minimal privileges, avoiding access to sensitive data such as login information, limiting internet access to an allowlist of domains, and human confirmation for decisions with real-world consequences such as financial transactions or agreeing to terms of service. It describes classifiers that scan screenshots for potential prompt injections and steer the model to check with the user, and says the other precautions remain important with the classifiers in place.
• Google: the Gemini API computer use documentation recommends running the agent in a sandboxed VM or container or a dedicated browser profile. Each proposed action can carry a safety decision; when it is require_confirmation, the application must ask the end user and send a safety acknowledgement. Built-in policy categories include financial transactions, sensitive data modification and account creation, and an opt-in setting enables prompt injection detection on screenshots.
• OpenAI: the computer use guide recommends a fresh browser profile or isolated desktop, limited network access with an allow list of sites and actions, confirmation for purchases, data transmission and destructive changes, step, time or cost limits, and checking the real outcome rather than relying on the model's final answer. It states that text in a page, document or tool result cannot grant permission or override the user's instructions.

Provider safeguards are a layer, not the boundary. The boundary is the environment you build: a disposable VM or container with no route to internal networks unless the task needs one, no shared clipboard or file system with a real workstation, egress through a proxy that enforces the allowlist and logs every request, and credentials injected for the session rather than stored in a profile. See the AI Agent Sandboxing and Code Execution Guide.

API agents need isolation too, but of a different kind: keep tokens in a broker or the tool server rather than in the model context, and run the agent runtime with no ambient cloud credentials it does not need.

Auditability

An API-integrated agent produces structured records: which client, on whose behalf, which tool, which arguments, which resource, which result. Those map directly onto NIST SP 800-53 AU-12 audit record generation and are easy to search. A computer-use agent produces screenshots and coordinates in the agent harness, while the target applications log the session's user, usually indistinguishable from a person. To make computer use auditable, keep the harness log of every screenshot and action tied to a task ID, use a dedicated agent account so application logs can separate agent actions from human ones, and record each approval with the exact action approved. See Agent Observability vs Audit Logging.

Decision guidance

Small teams often reach for computer use because it needs no integration work. That is reasonable for a pilot against low-risk applications, provided the environment is isolated and the account is dedicated. Large organisations should require an exception record for any production computer-use workflow, naming the system without an API, the owner and the date by which an integration will replace it.

Worked scenario: invoice processing

A finance team wants an agent to read supplier invoices from a shared mailbox, match them to purchase orders in the ERP system and queue payments. The first prototype uses computer use: it signs into webmail and the ERP web interface as a finance analyst. In review, the security team finds that the agent's session can approve payments up to the analyst's limit, that any invoice PDF can carry text the agent will read as instructions, and that ERP logs show only the analyst's name. The redesign moves mail and ERP access to APIs: a read-only mail token for one mailbox, an ERP integration that can create draft payment entries but not approve them, and a gateway policy blocking changes to supplier bank details. Approval stays with a human, bound to the exact amount and payee. One legacy supplier portal has no API, so a computer-use agent remains for that step only, in a disposable VM with a dedicated read-only portal account and an allowlist of that one domain. See the Invoice Redirection and Supplier Bank Detail Fraud Guide.

Implementing the interface choice

1. Inventory agent workflows: list each workflow, the systems it touches and whether each system has an API or MCP server.
2. Default to APIs: design new workflows against APIs; record an exception for any computer-use step with the reason and an exit date.
3. Build the computer-use environment once: a hardened image for disposable VMs or containers, an egress proxy with allowlists, no shared storage with workstations, and session teardown after each task.
4. Issue dedicated identities: separate agent accounts for computer use and scoped OAuth clients for API agents, each with an owner in the NHI inventory.
5. Add approval gates: confirmation for purchases, data transmission, destructive changes and consent prompts, enforced in the harness or gateway rather than by instruction alone. See the Human-in-the-Loop for AI Agents Guide.
6. Pilot in observe mode: run with approval on every action first, review logs, then relax approval for proven low-risk actions.
7. Keep rollback simple: be able to revoke the agent account or OAuth client and destroy the environment in one step.

Detection and measurement

• What to log: for API agents, every tool call with client, subject, scope, resource and decision; for computer-use agents, every screenshot and action with task ID, every egress request and every approval.
• Alerts: egress attempts to domains outside the allowlist, agent accounts signing in from unexpected environments, and tokens presented to the wrong audience.
• Computer-use share: percentage of production agent actions performed through screens rather than APIs, trending down.
• Exception age: number of computer-use exceptions past their exit date.
• Blocked actions: count of gateway denials and allowlist blocks per workflow, reviewed weekly for injection attempts.
• Approval quality: share of approvals where the reviewer saw the exact action (amount, recipient, record) rather than a summary.

Framework mapping

Common failure patterns

• Running in a person's own profile: the agent inherits every signed-in session and saved password on the workstation.
• Screens where an API exists: computer use chosen for speed of build against a SaaS application with a full API, giving up scopes and structured logs.
• Allowlist missing: the computer-use environment has open internet access, so injected instructions can send data anywhere.
• Approval by instruction only: the system prompt says "ask before paying", but nothing in the harness or gateway enforces it.
• One broad token: the API agent receives an administrator token or static key, recreating the blast radius of a full session.
• Shared agent accounts: several agents and people share one login, so logs cannot show who did what. See Shared credentials between AI agents.

Practitioner checklist

• Require an API or MCP integration by default and an exception record for each computer-use step.
• Run computer-use agents only in disposable VMs, containers or dedicated browser profiles.
• Give computer-use agents dedicated, minimal accounts, never a person's account.
• Enforce a domain allowlist through an egress proxy and log every request.
• Issue API agents audience-bound, short-lived, minimally scoped tokens, and forbid token passthrough.
• Enforce approval for purchases, data transmission, destructive changes and consent prompts in code.
• Keep screenshots, actions and approvals tied to a task ID for audit.
• Track computer-use share and exception age as programme metrics.

Standards and references

• Model Context Protocol: Authorization specification
• IETF: RFC 9700, Best Current Practice for OAuth 2.0 Security
• OWASP: Top 10 for LLM Applications 2025
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls
• Anthropic: Computer use tool
• Google: Gemini API computer use
• OpenAI: Computer use

Related NHI Mgmt Group resources: Browser and Computer-Use Agent Guide · Agentic Browsers Security Guide · Tool-Call Security for AI Agents · Prompt Injection Guide