How do security teams decide when to rely on model resistance versus runtime policy controls for AI agents?

Why This Matters for Security Teams

For AI agents, the question is not whether the model is “smart enough” to resist manipulation. The real issue is whether a compromised prompt, poisoned context, or deceptive tool output can still trigger a harmful state change. Model resistance can reduce successful attacks over time, but runtime policy controls are what stop an agent from spending money, exfiltrating data, deleting records, or minting credentials when the model is fooled.

This is why current guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework treats autonomy as a governance problem, not just a model-quality problem. NHIMG research on Gemini AI Breach — Google Calendar Prompt Injection and Replit AI Tool Database Deletion shows the practical risk: once an agent has execution authority, a single bad decision can propagate quickly across tools and systems.

In practice, many security teams discover that model resistance was never the controlling layer only after an agent has already taken an unsafe action.

How It Works in Practice

The decision should start with a simple principle: use model resistance to lower attack success rates, and use runtime policy controls to enforce hard boundaries. Model resistance includes prompt-injection training, instruction hierarchy tuning, sandboxing, and guardrail prompts. These measures can make an agent more robust against obvious manipulation, but they remain probabilistic. Runtime controls are deterministic enforcement points that evaluate each requested action before it executes.

That means the policy layer should decide whether an agent can call a tool, access a dataset, send an email, write to production, or request a secret. For agents, the strongest pattern is intent-based or context-aware authorisation: the request is evaluated at runtime based on what the agent is trying to do, what tool it is invoking, what data is in scope, and whether the action is allowed in the current business context. The agent may propose the action, but policy must approve it independently.

• Use model resistance to reduce prompt injection and tool misuse attempts.
• Use NIST Cybersecurity Framework 2.0 style governance to assign ownership and response paths.
• Use policy-as-code at the tool boundary so approval happens per request, not per session.
• Issue short-lived credentials and workload identity only when the agent needs them, rather than maintaining standing access.

That is also the core lesson in NHIMG coverage such as OWASP NHI Top 10 and LLMjacking: How Attackers Hijack AI Using Compromised NHIs: once credentials or tool access are available, a fooled model can become an execution path for an attacker. These controls tend to break down when agents can chain tools across loosely governed services because the policy engine no longer sees the full intent chain in one place.

Common Variations and Edge Cases

Tighter runtime policy often increases operational overhead, requiring organisations to balance safety against latency, developer friction, and false positives. That tradeoff is real, especially in fast-moving agentic environments where teams want autonomy but still need reviewable control points. Best practice is evolving, and there is no universal standard for how much model resistance is “enough” before policy must take over entirely.

In high-trust internal workflows, teams sometimes accept stronger model resistance for low-impact tasks such as drafting, summarisation, or research, while reserving strict runtime controls for anything that can mutate state or expose secrets. In customer-facing or production-adjacent agents, the balance shifts sharply toward deterministic controls. The most important distinction is not whether the model is open-source or closed, but whether a failed model can still cause damage. If yes, runtime policy must remain the final gate.

This is especially important when agents have access to secrets, payment systems, admin consoles, or code deployment pipelines. NHIMG reporting on The State of Secrets in AppSec shows how fragile secret handling already is in ordinary environments; agentic systems amplify that risk because access can be requested dynamically and at machine speed. For that reason, security teams should treat model resistance as a resilience layer and runtime policy as the enforcement layer, not as substitutes for each other.

Related resources from NHI Mgmt Group

• How should security teams manage permissions for AI agents?
• How should security teams govern AI agents that use OAuth access?
• How should security teams limit the risk from AI agents that have access to production systems?
• How should security teams govern AI agents that can access enterprise systems?