Runtime testing evaluates a live application or a running build to see how security controls behave under actual execution. Pre-deployment code testing inspects source or compiled artifacts for possible defects before they are exercised. Runtime testing is better for confirming exploitability, business logic issues, and authorization behavior. Pre-deployment testing is better for earlier feedback and broader code coverage.
Why This Matters for Security Teams
The distinction matters because these two test modes answer different questions. Pre-deployment code testing helps teams find weaknesses before release, when fixes are usually cheaper and less disruptive. Runtime testing shows whether a control, exploit path, or authorization decision actually behaves as expected in a live context. For security teams, confusing the two often creates a false sense of coverage: code can look clean while the running service still fails open, trusts the wrong input, or exposes a broken permission boundary.
That difference is especially important when applications depend on configuration, identity state, APIs, or dynamic policy enforcement. A static scan may identify risky patterns, but only runtime validation can confirm whether those patterns become reachable in production. The NIST Cybersecurity Framework 2.0 reinforces the need to manage risk across build and operational phases, not just during development.
In practice, many security teams encounter the real weakness only after a control has already been bypassed in production, rather than through intentional pre-release validation.
How It Works in Practice
Pre-deployment testing usually includes source code analysis, dependency scanning, software composition analysis, unit and integration security tests, and sometimes static analysis of compiled artifacts. These methods are strong at finding insecure patterns early, such as injection-prone input handling, unsafe deserialization, weak cryptography, or missing validation. They are also valuable in CI/CD pipelines because they create fast feedback and support broader coverage across large codebases.
Runtime testing, by contrast, observes the application while it is executing. It can include dynamic application security testing, authenticated functional security checks, interactive testing, environment-specific probes, and controlled exploitation in a staging or production-like context. Runtime validation is often the only way to confirm whether controls actually work when real requests, identities, sessions, data flows, and policy decisions are involved. This matters for business logic flaws, authorization bypass, session handling, anti-automation controls, and state-dependent vulnerabilities.
- Use pre-deployment testing to catch defects before they become operational risk.
- Use runtime testing to verify exploitability and control behavior in realistic conditions.
- Combine both when the application depends on complex identity, API, or workflow logic.
- Feed findings back into development, configuration, and release gates.
For organisations aligning with secure software practice, OWASP Top 10 is a useful lens for what tends to be found late, while runtime-focused validation can expose gaps that static analysis cannot prove or disprove on its own. These controls tend to break down when environments are highly distributed and heavily dependent on third-party services because test conditions diverge from production behaviour.
Common Variations and Edge Cases
Tighter testing often increases delivery overhead, requiring organisations to balance release speed against confidence in control effectiveness. Best practice is evolving here, especially for systems that change rapidly or rely on ephemeral infrastructure, AI-generated code, or complex identity chains.
Some teams treat “testing before deployment” as a single activity, but that is too coarse. A build may be scanned successfully and still fail once deployed because runtime context changes the outcome. Container images, serverless functions, feature flags, service meshes, and external APIs can all alter how a security issue manifests. That is why current guidance suggests pairing pre-release controls with runtime monitoring and verification rather than choosing one or the other.
The tradeoff becomes sharper in regulated or high-availability environments. Runtime testing can be constrained by uptime requirements, data sensitivity, or fear of impacting users, while pre-deployment testing can miss issues that only appear with live authentication, caching, concurrency, or policy evaluation. For this reason, many teams use pre-deployment testing for breadth and runtime testing for proof. The practical goal is not perfect coverage in one phase, but confidence that both code defects and live execution risks are being addressed. The CISA secure software development guidance is useful for structuring that separation of responsibilities.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.RM-01 | Risk decisions should reflect both pre-release and live control validation. |
| OWASP Agentic AI Top 10 | Agentic systems need runtime checks for tool use, policy enforcement, and action safety. | |
| NIST AI RMF | Map | AI systems need lifecycle testing across development and deployment phases. |
| MITRE ATLAS | Adversarial AI testing often requires runtime observation of model behaviour. | |
| NIST AI 600-1 | GenAI systems need validation of both prompt handling and deployed behaviour. |
Treat build-time and runtime testing as linked risk signals in your security governance process.
