They assume instructions inside the agent are durable controls. In reality, any policy text stored where the agent can edit it can be weakened, bypassed, or rewritten by the same system it is supposed to constrain. Durable governance needs an external control layer that the agent cannot modify.
Why This Matters for Security Teams
In-context governance is attractive because it feels immediate: the agent sees the rule, follows the rule, and the system appears controlled. The problem is that prompt text, memory content, and tool-facing instructions are not the same as enforceable policy. If an agent can rewrite, summarise, retrieve, or prioritise its own guidance, then the control surface becomes part of the attack surface. That is exactly why the NIST AI Risk Management Framework treats governance as a lifecycle discipline, not a prompt engineering trick.
Security teams often underestimate how quickly instruction drift appears once an agent is connected to retrieval, tools, or long-running memory. A well-written policy can be bypassed by conflicting context, poisoned knowledge, or a tool response that the agent treats as authoritative. The issue is not only malicious compromise. Ordinary operational changes such as new workflows, updated permissions, or expanded tool access can silently invalidate the original guardrails. In practice, many security teams encounter governance failure only after an agent has already taken a harmful action, rather than through intentional policy enforcement.
How It Works in Practice
Effective governance for agents needs a layered design. The agent may hold instructions, but the actual decision to permit, deny, redact, escalate, or log should sit outside the agent in a policy enforcement layer that the agent cannot modify. That external layer should evaluate the requested action, the user context, the tool scope, and the current risk posture before any execution occurs. This aligns well with the control thinking behind the NIST Cybersecurity Framework 2.0 and the attack-pattern view in the MITRE ATLAS adversarial AI threat matrix.
In operational terms, teams should separate three things:
- Policy intent, which defines what the agent should never do.
- Execution control, which decides whether a tool call or data access is allowed.
- Audit evidence, which records what happened and why.
That separation matters because agentic systems can be manipulated through prompt injection, memory contamination, or tool-output abuse. The current guidance suggests treating retrieved content and conversational state as untrusted inputs, not as a source of policy truth. It is also wise to keep high-risk actions behind step-up approval, especially where the agent can change permissions, move money, send messages, or modify records. If the system uses multiple models or orchestrators, governance should be applied at the orchestration boundary rather than inside individual prompts. The operational goal is to make policy durable even when the agent’s context changes.
This approach should be paired with model and workflow monitoring so that security teams can detect when an agent begins to reason outside expected boundaries. The OWASP Top 10 for Agentic Applications 2026 and the CSA MAESTRO agentic AI threat modeling framework both reinforce the need to threat model tool use, autonomy boundaries, and prompt-driven escalation paths. These controls tend to break down when legacy workflows allow the agent to directly edit the same memory, policy, or configuration store that is supposed to constrain it.
Common Variations and Edge Cases
Tighter governance often increases latency and operational overhead, requiring organisations to balance autonomy against control. That tradeoff is real, especially in customer support, software engineering, and SOC workflows where teams want fast agent responses but cannot afford uncontrolled action. Best practice is evolving, but there is no universal standard for how much policy should live inside the agent versus outside it.
One common edge case is the “self-updating assistant,” where the agent can write back into its own instructions or retrieve them from a mutable knowledge base. That pattern usually looks efficient until a poisoned or misleading update changes future behaviour. Another is delegated tool use, where an agent can create tickets, trigger workflows, or call admin APIs. In those environments, instruction text alone is too weak because the real risk sits in authority amplification, not wording. The same caution applies to systems that mix human approvals with partial automation: if the agent can draft the approval context, it may bias the reviewer without technically violating the rule.
For higher-risk deployments, use external policy checks, immutable audit logs, and separate administrative control planes. Where the agent handles regulated data or critical decisions, governance should also be mapped to documented risk management under NIST AI Risk Management Framework principles and, for security posture, the NIST Cybersecurity Framework 2.0. The practical test is simple: if the agent can change the rule, the rule is not durable governance.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | Agentic app risks center on prompt injection, tool abuse, and self-modifying instructions. | |
| NIST AI RMF | GOVERN | Govern function fits durable oversight, accountability, and policy control for AI systems. |
| NIST CSF 2.0 | PR.AC-4 | Least-privilege access is central when agents can invoke tools or modify records. |
| MITRE ATLAS | ATLAS covers adversarial tactics like prompt injection and model manipulation. | |
| CSA MAESTRO | MAESTRO addresses threat modeling for autonomous agent workflows and trust boundaries. |
Assign ownership, approval, and monitoring outside the model before granting tool authority.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 18, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
