Pre-deployment testing looks for weaknesses before an agent is released, using adversarial scenarios to expose unsafe behaviour. Runtime security protects the agent while it is operating, monitoring live inputs, outputs, and actions for abuse or drift. Organisations need both because some failures only appear under production conditions, especially when agents interact with real users and systems.
Why This Matters for Security Teams
Pre-deployment testing and runtime security solve different failure modes, and treating them as interchangeable creates blind spots. Testing before release can uncover prompt injection paths, tool misuse, and unsafe output patterns, but it cannot reproduce every production interaction. Runtime controls are needed once an agent is connected to live data, users, and systems, where behaviour can change with context. Guidance from the NIST AI Risk Management Framework is clear that AI risk must be managed across the full lifecycle, not just at launch.
This distinction matters even more for agentic systems because execution authority is the risk surface. An agent can chain tools, request secrets, and take actions that static tests may not anticipate. NHIMG research on OWASP Agentic Applications Top 10 shows why agent-specific risks require both design-time scrutiny and live monitoring. In practice, many security teams encounter these failures only after an agent has already touched a real tenant, data store, or production workflow, rather than through intentional pre-release validation.
How It Works in Practice
Pre-deployment testing is about proving the agent can be built and configured safely before it is allowed to operate. That usually includes adversarial prompt testing, tool-call abuse cases, policy checks, red-team simulations, and validation of the agent’s planned permissions. The goal is to catch predictable weaknesses in prompts, orchestration logic, retrieval paths, and secret handling. For agentic systems, frameworks such as the CSA MAESTRO agentic AI threat modeling framework and NHIMG’s Analysis of Claude Code Security are useful because they focus on the places where autonomous behaviour meets real permissions.
Runtime security begins after deployment and watches what the agent is actually doing. That means live inspection of prompts, context, tool calls, outputs, and downstream actions, with policy decisions made at request time. Current best practice is evolving toward intent-aware controls, short-lived credentials, and workload identity so the agent can be authorised for a specific task rather than granted broad standing access. That aligns with the OWASP Top 10 for Agentic Applications 2026 and the MITRE ATLAS adversarial AI threat matrix, both of which emphasise runtime abuse, chaining, and escalation risks.
- Pre-deployment testing validates prompts, tools, guardrails, and planned access before release.
- Runtime security enforces policy on live requests, outputs, and side effects while the agent is operating.
- JIT credentials and short TTLs reduce exposure when an agent needs temporary access to a system or secret.
- Monitoring must include tool chaining, data exfiltration attempts, and privilege escalation paths.
These controls tend to break down in highly dynamic environments such as customer-facing copilots with broad SaaS integrations because the live context changes faster than static test cases can cover.
Common Variations and Edge Cases
Tighter runtime control often increases latency, operational overhead, and tuning effort, so organisations must balance safety against workflow friction. That tradeoff is especially visible when an agent needs to complete multi-step work across several systems, because each step may require a new policy decision. There is no universal standard for this yet, so current guidance suggests matching control strength to the agent’s autonomy, data sensitivity, and blast radius.
One common edge case is when pre-deployment testing passes but runtime still fails because the agent sees unexpected user content, a new tool response, or a novel sequence of actions. Another is overly permissive standing access: static RBAC can describe who the agent should be, but it cannot fully capture what the agent is trying to do at a specific moment. NHIMG’s The State of Non-Human Identity Security highlights why this matters, especially where over-privileged accounts and weak rotation create easy paths for abuse. The practical answer is layered defence: test aggressively before release, then assume the agent will behave differently under production pressure and enforce runtime checks accordingly.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST Zero Trust (SP 800-207) set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A1 | Covers prompt injection and agent abuse that testing and runtime must both address. |
| CSA MAESTRO | TA-03 | Addresses agent threat modeling across design and execution phases. |
| NIST AI RMF | GOVERN | Requires lifecycle AI risk governance, not just pre-launch assurance. |
| OWASP Non-Human Identity Top 10 | NHI-03 | Runtime security depends on ephemeral secrets and rotation for agent identities. |
| NIST Zero Trust (SP 800-207) | PR.AC-4 | Zero trust supports runtime authorization based on context and least privilege. |
Use short-lived agent credentials and rotate or revoke them automatically after task completion.
Related resources from NHI Mgmt Group
- How do security teams balance pre-deployment testing and runtime validation for AI systems?
- What is the difference between SAST tools and runtime security tools for AI coding agents?
- What is the difference between AI code reasoning and runtime security testing?
- What is the difference between pre-deployment scanning and runtime protection?
