Human-Agent-System Model

How the human-agent-system model works

The human-agent-system model separates the request path into three distinct roles. The human sets intent, the agent interprets and acts, and the system decides what is accepted, enforced, or blocked. That separation is what makes the model useful for agentic workflows: it forces you to examine where authority actually changes hands.

In practice, the model helps distinguish delegation from execution. A human may approve a task, but the agent can still overstep if its permissions, instructions, or tool access are broader than the human intended. A system can also enforce stronger rules than the agent expects, which is why the model is as much about boundaries as it is about autonomy.

Why the model matters for consent and control

The value of the model is that it exposes three different control points. Consent belongs to the human, behaviour belongs to the agent, and enforcement belongs to the system. If those three are not aligned, an apparently authorised action can still become unsafe, excessive, or misleading.

This is especially important in workflows where a human is “in the loop” but not necessarily in control of each step. The human may approve a goal once, while the agent chains many actions under that approval. The system may accept those actions because the guardrails are too coarse, or because policy is checked only at the start rather than at each action. That is the core weakness the model is designed to reveal.

For deeper context on how agent behaviour changes as autonomy increases, see AI Agents vs Agentic AI. For identity and delegation patterns behind acting on behalf of a human, Agentic AI Identity Guide is a useful companion.

Common failure modes in agentic workflows

The main failure mode is drift between the human’s intent and the agent’s actual actions. That can happen when the agent has standing access, reuses a prior session, inherits broad credentials, or interprets a prompt more liberally than the human intended. It can also happen when the system treats a delegated request as if it were a direct user action.

Another failure mode is hidden privilege expansion. An agent may begin with a narrow task, then use tools, connectors, or inherited tokens that let it reach data or functions the human did not explicitly approve. If the system does not re-check policy at meaningful points, the request can look legitimate while still violating the intended boundary.

These issues are not limited to theory. Controls for per-action authorisation, delegation, and least privilege are central to safe agent operation, which is why practical guidance on AI Agent Authorisation Guide and containment patterns in Zero Trust for AI Agents map well to the model. For a broader threat view, Agentic AI Security Guide shows how identity, tools, and orchestration interact.

How to interpret the model in security architecture

Architecturally, the model is a way to draw trust boundaries. It tells you where to ask, “Who requested this?”, “Who executed it?”, and “Who allowed it?”. Those are different questions, and each one can have a different answer in an agentic system.

That makes the model useful for design reviews, threat modelling, and control placement. If the human approves a task, the agent executes it, and the system enforces it, then each layer needs its own policy, logging, and revocation model. The model is most valuable when it prevents a false assumption that one approval automatically covers all downstream actions.

In mature deployments, this usually means designing for explicit delegation, auditable action boundaries, and system-side enforcement that does not rely on the agent’s own self-restraint. Resources such as AI Agent Observability, Audit and Incident Response Guide help with attribution and response when the human-agent-system chain goes wrong.

Risk and Threat Considerations

The security risk in this model is not the existence of three parties, but the gaps between them. If human approval, agent behaviour, and system enforcement are not tightly aligned, attackers can abuse delegated trust, overbroad access, or weak action controls to turn a legitimate workflow into an execution path for unintended actions.

Failure mechanism: The agent acts with more authority than the human intended, or the system accepts actions without re-validating scope, so delegated intent expands into unintended access, data movement, or tool use.

Impact: That can lead to privilege abuse, unauthorized actions, audit confusion, and harder incident response because the resulting activity may appear user-approved even when it exceeded the intended delegation.

Practitioner Guidance

Why practitioners should care: The model is most useful when it changes how you think about control ownership. Humans own intent, agents own execution behaviour, and systems own enforcement. If a control only exists in one of those layers, assume the other two can still fail in ways that matter operationally.

Common misunderstanding: “Human approval” is often treated as a sufficient control by itself. In agentic workflows, approval is only one input to a broader authority chain, so practitioners should treat it as necessary but not complete.

Practitioner takeaway: Design reviews should ask where delegation starts, where action is checked, and where enforcement happens, because the model only works when those three answers are explicit and consistent.

Related resources from NHI Mgmt Group

• How should teams design an agent harness so the model can act safely without constant human configuration?
• What breaks when AI agent sandboxes rely on a human approval model?
• What happens when incident response is handled by a multi-agent system without human governance?
• What is the difference between a model that answers questions and an agent system that can act on business systems?