Computer use is the ability of an AI system to operate software through the user interface the way a person would. It includes clicking, typing, opening files, reading screens, and moving between applications when APIs are unavailable or incomplete.
Expanded Definition
Computer use describes an AI system interacting with a graphical user interface, browser, desktop, or business application through visible controls rather than a direct API. The key boundary is that the model is not merely generating text about a task; it is taking the actions that a user would normally perform on screen.
This matters because computer use is a fallback capability, not a synonym for automation in general. When APIs exist, they are usually more predictable, auditable, and easier to constrain. Computer use fills the gap when systems expose only partial integrations, legacy interfaces, or inconsistent workflow support. The trade-off is that screen-based execution is more flexible but also more brittle, because layout changes, pop-ups, timing issues, and ambiguous page states can alter behavior.
In current guidance, there is consensus that computer use is best treated as a distinct execution mode with its own trust boundaries and supervision needs. NHI Management Group views the practical distinction as simple: if the AI is navigating the interface as an operator would, it should be governed as a higher-uncertainty control path than direct machine-to-machine integration.
Examples and Use Cases
Computer use appears anywhere a system must complete a workflow without stable programmatic access. Common examples include:
- An AI assistant opens a customer portal, reads the page, and submits a form when no API is available.
- A support workflow uses browser navigation to retrieve records from a legacy application that exposes only a GUI.
- An agent compares two on-screen dashboards, copies values between applications, and updates a ticketing record.
- A finance or operations assistant interacts with web-based tools that require multi-step clicking and manual confirmation.
- A helpdesk workflow uses a browser session to carry out routine account maintenance across fragmented systems.
The main implementation trade-off is reliability versus reach. Screen-driven operation can extend automation into older or proprietary systems, but it typically needs tighter supervision because the AI must infer state from pixels or rendered text rather than from structured responses. Where workflows are high-value or safety-sensitive, many teams reserve computer use for narrow, well-bounded tasks and keep irreversible steps behind human review.
Security Implications
Computer use changes the risk profile of AI by giving the system operational reach inside real user interfaces. That expands the blast radius of a prompt injection, misleading screen content, poisoned webpage, or mistaken action sequence, because the model can click, submit, download, or approve actions that have real effects. It also makes verification harder, since the system may appear to succeed while interacting with the wrong window, the wrong account, or an altered page state.
Another common failure mode is over-trust in what the interface shows. A page can display incomplete context, manipulated copy, or hidden controls that steer the model into unsafe choices. In practice, this means computer use should be assumed to carry higher operational uncertainty than API-based automation, especially when it can access financial, administrative, or data-bearing applications.
For NHIMG readers, the important observation is that the security problem is not just “the model can act.” It is that the model can act through a human-style interface where identity, session state, and authorization are already implicit, making misuse harder to detect and easier to confuse with legitimate work.
Domain and Governance Relevance
Computer use belongs primarily to AI security and operational governance, because it is about how an AI system executes work in the real world. The governance question is whether the task justifies screen-level autonomy, what limits apply to those actions, and how much supervision is needed when the system must infer state from a user interface rather than from a structured integration.
The identity dimension becomes material when the AI operates inside authenticated sessions, shared workstations, or delegated business applications. In those cases, the control issue is not only what the model can see, but what it can do under an existing user context. That is where human access governance and machine action governance begin to overlap, especially for systems that can approve transactions, modify records, or move across applications.
For practitioners, the core boundary is that computer use should not inherit human trust assumptions by default. The fact that a UI is familiar does not make an AI operator equivalent to a person, and that distinction should shape logging, review, and authorization decisions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS address the attack surface, NIST AI RMF, CIS Controls v8 and NIST CSF 2.0 set the technical controls, and ISO/IEC 42001:2023 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | MAP — Measure, Assess, and Manage | Computer use creates AI execution risk that should be measured and managed. |
| Recommendation — Measure UI-driven agent behavior and manage the added uncertainty before expanding autonomy. | ||
| ISO/IEC 42001:2023 | 6.1 — Actions to Address Risks and Opportunities | Computer use requires AI governance decisions about when screen-based action is acceptable. |
| Recommendation — Define when computer use is permitted, supervised, or prohibited for each workflow. | ||
| CIS Controls v8 | 6 — Access Control Management | UI-driven AI actions often run inside authenticated sessions and need tight access control. |
| Recommendation — Restrict session scope and review UI actions under the least-privilege principle. | ||
| NIST CSF 2.0 | PR.AA — Identity Management, Authentication, and Access Control | Computer use can operate within human-authenticated contexts that require access governance. |
| Recommendation — Enforce authentication and access boundaries before allowing AI to act in live applications. | ||
| MITRE ATLAS | AML.T0015 — Prompt Injection | Prompt injection can steer an AI operating through a GUI into unsafe actions. |
| Recommendation — Test computer-use workflows for prompt injection paths that alter tool use and outcomes. | ||
Related resources from NHI Mgmt Group
- How do you know if computer-use governance is actually working?
- How should security teams govern computer-use models that change access inside enterprise systems?
- How can organisations decide whether a computer-use model belongs in production IAM?
- Who should approve computer-use automation for privileged identity tasks?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 9, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
