What breaks when organisations rely only on pre-deployment testing for agentic AI security?

Why This Matters for Security Teams

Pre-deployment testing is necessary, but it is not sufficient for agentic ai security because the system’s real risk emerges when an agent is allowed to act, chain tools, and interpret fresh context. That gap matters most when the agent can access files, send messages, change records, or trigger workflows. Guidance from the NIST AI Risk Management Framework treats this as an ongoing governance problem, not a one-time validation exercise.

Teams often overestimate safety because a model behaved correctly in a controlled test set. In production, prompt injection, malicious instructions embedded in retrieved content, and indirect data poisoning can turn a compliant agent into a risky operator. The control question is not only whether the agent can answer well, but whether it can resist manipulation while acting under changing conditions. The practical failure is that security reviews focus on the model boundary while the real attack surface sits in the orchestration layer, tool permissions, and runtime context. In practice, many security teams encounter harmful agent actions only after an external input or internal workflow has already been abused, rather than through intentional runtime detection.

How It Works in Practice

agentic ai security needs layered controls that extend beyond model evaluation into runtime monitoring, access restriction, and action approval. Pre-deployment tests can validate baseline behavior, but they cannot fully predict how an agent will behave when it encounters new prompts, retrieved documents, or tool responses. Current guidance from the OWASP Agentic AI Top 10 and the CSA MAESTRO agentic AI threat modeling framework points to the same operational truth: security has to follow the agent into production.

Effective programs usually combine these controls:

• Constrain tool access so the agent only reaches the systems it truly needs.
• Validate inputs and retrieved content for prompt injection and malicious instructions.
• Require approval or step-up checks before high-impact actions are executed.
• Log prompts, tool calls, outputs, and policy decisions for review and incident response.
• Continuously evaluate drift, because behaviour can change as context, memory, or workflows evolve.

This is also where adversarial testing matters. MITRE ATLAS adversarial AI threat matrix is useful for mapping likely attack paths, while runtime policies help catch abuse that synthetic tests missed. For higher-risk deployments, teams should treat agent permissions like privileged access, with explicit scoping and revocation paths. These controls tend to break down when the agent is embedded in fast-moving workflow automation with broad tool access and weak logging because the system can act faster than human review can intervene.

Common Variations and Edge Cases

Tighter runtime control often increases friction and latency, requiring organisations to balance autonomy against operational safety. That tradeoff is especially visible in customer-facing agents, internal copilots with broad search access, and agents that write to business systems. Best practice is evolving, and there is no universal standard for how much autonomy is acceptable without human approval.

Some environments can tolerate more pre-deployment testing, but only when the agent has no direct execution authority or operates in a tightly bounded sandbox. Once the system can send emails, update records, execute code, or retrieve sensitive data, testing alone becomes a weak assurance signal. The risk also rises when retrieval-augmented generation is involved, because poisoned documents or malicious content can influence decisions after launch. Practitioners should also distinguish between model safety and system safety: a model can be aligned while the surrounding agent workflow remains exploitable.

For that reason, NHI Management Group recommends pairing model-level evaluation with operational controls, observability, and incident response playbooks. The point is not to eliminate change in production, but to make unsafe action visible and stoppable. Where that visibility is missing, agentic systems can pass every pre-launch check and still fail the moment they encounter adversarial context in the real world.

Related resources from NHI Mgmt Group

• What breaks when AI controls stop at pre-deployment testing?
• What breaks when organisations rely on manual data classification for AI security?
• What breaks when organisations rely on static scans for AI security?
• What breaks when organisations rely on static security testing?