How should teams govern AI agents without treating the model as the only control point?

Why This Matters for Security Teams

AI agents create a governance problem that looks familiar at first, then fails in a different place. The model may be versioned and tested, but the real control surface includes prompts, tools, memory, retrieval sources, sub-agent delegation, and the approval path for each action. That is why guidance such as the NIST AI Risk Management Framework matters: it frames AI risk as an end-to-end lifecycle issue, not a checkpoint exercise.

For security teams, the practical risk is that an agent can behave safely in evaluation and still become unsafe once it is connected to live credentials, production data, or external APIs. The same model can produce very different outcomes when the harness changes. Prompt injection, tool abuse, data exfiltration, and overbroad delegation are governance failures as much as technical failures, because they arise from weak policy binding across the runtime. Current guidance suggests that agent governance must include approval gates, trace logging, and least-privilege tool access if it is to be meaningful.

In practice, many security teams encounter agent misuse only after the tool chain has already been trusted and deployed without intentional runtime control design.

How It Works in Practice

Effective governance treats the agent as a system of recordable decisions, not a single model call. The policy question is not only, “Can this model answer?” but also, “Can this agent call this tool, in this context, with this data, under this approval rule?” That is where control design shifts from model-centric review to runtime enforcement. The OWASP Agentic AI Top 10 is useful here because it highlights agent-specific failure modes such as excessive agency, unsafe tool invocation, and weak output handling.

• Define the agent boundary: model, tools, memory, retrieval, and human approvals.
• Version control prompts, policies, tool manifests, and workflow logic together.
• Separate sub-agents by role and data access, rather than letting them inherit broad context.
• Log every governed run with inputs, decisions, tool calls, outputs, and overrides.
• Require step-up approval for high-risk actions such as sending data, changing permissions, or executing code.

Traceability matters because post-incident reconstruction depends on knowing which instruction, retrieval result, or tool output drove the action. This is also where control mapping to NIST Cybersecurity Framework 2.0 becomes practical: identify, protect, detect, respond, and recover all need agent-specific instrumentation. For threat modelling, the MITRE ATLAS adversarial AI threat matrix helps teams reason about prompt injection, poisoning, and abuse paths in a structured way. These controls tend to break down when agents are allowed to self-chain across multiple tools in low-latency production workflows because approvals, logs, and policy checks get bypassed for speed.

Common Variations and Edge Cases

Tighter runtime governance often increases latency and operational overhead, so organisations have to balance assurance against workflow friction. That tradeoff becomes most visible in customer-facing agents, developer copilots, and autonomous back-office automations where too much gating can undermine usefulness. Best practice is evolving here, and there is no universal standard for exactly how much autonomy is acceptable in each case.

One common edge case is the “safe model, unsafe wrapper” problem: the model passes review, but a later change to tools, retrieval data, or approval routing introduces risk without a new assessment. Another is shared memory across sub-agents, which can create accidental cross-contamination of sensitive context. The governance answer is usually to classify the runtime components separately and to re-approve the system when any component materially changes. Where agents touch regulated data or critical operations, teams should also align governance with the CSA MAESTRO agentic AI threat modeling framework and apply the same discipline to delegated actions that they already expect for privileged access. For deeper attack-pattern analysis, the Anthropic report on AI-orchestrated cyber espionage is a reminder that agents can be operationalised for abuse as quickly as they can be operationalised for productivity.

Related resources from NHI Mgmt Group

• How should security teams govern AI agents without creating a manual review bottleneck?
• How should security teams govern AI agents that use Model Context Protocol?
• How should security teams govern AI agents using Model Context Protocol?
• How should security teams govern AI voice agents that chain multiple model calls?