What breaks when security teams rely only on human-centric threat models for AI agent activity?

Why This Matters for Security Teams

Human-centric threat models assume attackers progress in familiar, reviewable steps: one login, one escalation, one move, one alert. AI agents do not necessarily behave that way. They can chain tool calls, retry with variation, and maintain context across sessions, which means a narrow analyst view can miss the attack path even when every individual action looks ordinary. Current guidance from the NIST AI Risk Management Framework is clear that AI risk must be handled as a system property, not just a user behavior problem.

The practical failure is often one of telemetry interpretation. A SIEM or SOAR workflow tuned to human adversaries may flag only obvious misuse, while an agent quietly performs reconnaissance, secrets discovery, or authorization probing through legitimate interfaces. That gap becomes larger when the agent has access to APIs, workflow engines, and browser automation. Security teams then spend their time validating isolated alerts instead of understanding the autonomous sequence that produced them. In practice, many security teams encounter AI-driven abuse only after tool-level activity has already translated into privilege gain or data exposure, rather than through intentional detection of agent behavior.

How It Works in Practice

Effective defense starts by modeling the agent as an autonomous software entity with execution authority, not as a person using a keyboard. That changes the threat model in several ways: the attacker may abuse prompt injection, manipulate tool outputs, induce the agent to reveal secrets, or steer the system into unintended transactions. The attack surface is not limited to the model prompt. It includes connectors, retrieval layers, APIs, memory stores, and any identity used by the agent to act. Frameworks such as the MITRE ATLAS adversarial AI threat matrix and the OWASP Agentic AI Top 10 are useful because they move the conversation from “who clicked what” to “what behavior did the system enable.”

• Instrument tool use, not just prompts, so each external action is attributable and reviewable.
• Separate model inference from privileged execution paths, and require policy checks before tool invocation.
• Bind agent identities to least privilege, short-lived credentials, and explicit approval for sensitive actions.
• Log chain-of-action telemetry so investigators can reconstruct sequences, not isolated events.
• Validate outputs before execution when downstream actions affect money, data, or production systems.

Security teams should also consider whether an agent is operating under a user’s human identity, a service identity, or a dedicated non-human identity. That distinction matters for access reviews, incident response, and containment. The Anthropic report on the first AI-orchestrated cyber espionage campaign report is a strong reminder that autonomous activity can compress an attack cycle faster than manual controls were designed to withstand. These controls tend to break down when agents are embedded in high-trust automation pipelines because the business process itself becomes the attacker’s cover.

Common Variations and Edge Cases

Tighter agent oversight often increases operational overhead, requiring organisations to balance autonomy against governance friction. That tradeoff is real, but current guidance suggests it is safer than allowing unrestricted execution in production. Some environments can tolerate more flexibility, such as low-risk content drafting or internal summarisation, while others, such as finance, admin, or privileged infrastructure automation, need stricter action gating and human approval. There is no universal standard for this yet, so teams should classify agent tasks by blast radius and reversibility.

Edge cases usually appear when an agent crosses trust boundaries. A model may be safe in a sandbox but unsafe once connected to email, ticketing, code deployment, or cloud control planes. Similarly, an agent that appears benign in a single session may become dangerous when memory persists, retrieval is poisoned, or multiple tools are chained together. The CSA MAESTRO agentic AI threat modeling framework is useful here because it encourages control placement across the whole agentic workflow, not just the model endpoint. Teams should treat ambiguous behavior as a governance issue first and a detection issue second, especially when output validation, approval workflows, and identity binding are weak. For broader monitoring and incident triage, CISA cyber threat advisories remain a useful reference point for emerging tactics, but they do not replace agent-specific threat modeling.

Related resources from NHI Mgmt Group

• What breaks when security teams rely only on DSPM for AI agent governance?
• What breaks when merchants rely on human fraud models to classify AI agent traffic?
• What breaks when security teams track human and AI agent risk separately?
• What breaks when teams rely on a single generic filter for AI agent security?