Pre-Deployment AI Testing vs Runtime Monitoring: Where AI Assurance Breaks Down

Most AI assurance programmes put their effort into a release gate, then treat the passing result as if it still described the system months later, when the model, the inputs and the attackers have all moved on. Pre-deployment testing and runtime monitoring answer different questions, and each has blind spots the other has to cover. This guide maps those coverage gaps (unknown inputs, drift, provider model updates, adaptive attackers), shows what each discipline actually detects, sets out what regulators and NIST expect from each, and gives a way to split a limited assurance budget between them. It is written for AI security leads, AI governance owners and platform teams who already run some testing or some monitoring and need to decide where the next pound goes. Tooling choices are covered separately in the AI-SPM vs AI Runtime Security comparison.

Key takeaways

• Testing proves a claim about one configuration. A pre-deployment result is valid for the model version, prompts, tools, guardrails and data it ran against, and expires when any of them change.
• Monitoring sees the real world but only after it happens. It catches unknown inputs, drift and live attacks, yet it detects rather than prevents, and it only sees what you instrumented.
• The gaps are predictable. Testing misses unknown inputs, adaptive attackers and silent provider updates; monitoring misses rare high-impact failures, slow harms with no visible signal and anything outside the telemetry.
• Regulators expect both, linked. The EU AI Act requires testing before market placement against predefined metrics and a post-market monitoring system that feeds risk management; the NIST AI RMF says lab measurements may differ from real-world risk.
• The feedback loop is the control. Every confirmed runtime finding becomes a regression test, and every model or configuration change triggers a re-test, otherwise the two disciplines drift apart.

Short answer

Pre-deployment testing asks "does this exact system behave acceptably against the risks we know about, before anyone relies on it?" It runs in a controlled environment, against defined datasets and attacks, and produces a pass or fail decision with evidence. Runtime monitoring asks "is the system we actually run still behaving acceptably, for the users and inputs it actually meets?" It runs in production, against real traffic, and produces signals, alerts and trends.

Neither substitutes for the other. The NIST AI Risk Management Framework states the problem directly in its discussion of measurement: measuring AI risks in a laboratory or controlled environment can give useful insight before deployment, but those measurements may differ from the risks that emerge in operational, real-world settings. For the vocabulary of evaluations, red teaming and benchmarks, see AI Red Teaming vs Evaluations vs Benchmarks.

Side-by-side comparison

Pre-deployment testing in depth

Threat model

Testing assumes the risks can be named in advance. It targets known attack classes (direct and indirect prompt injection, tool misuse, sensitive data disclosure, jailbreaks), known quality failures (accuracy below the declared level, unfair outcomes for a group) and known unsafe actions for agents. MITRE ATLAS techniques such as AML.T0051 LLM Prompt Injection, AML.T0051.001 Indirect and AML.T0054 LLM Jailbreak give each test case a label; the EU AI Act Article 15(5) list (data poisoning, model poisoning, adversarial examples, confidentiality attacks, model flaws) is a useful coverage prompt.

Architecture in words

A harness loads versioned datasets, drives the system under test through the same prompts, tools and permissions it will have in production (backed by mocks, seeded canary records and fake credentials), scores the results, and writes a record that ties every score to the model identifier, prompt version, tool definitions, guardrail configuration and dataset version. Fast smoke suites run on each change; the full suite with repeated attempts runs on release candidates. The design and gating detail is in the AI Security Evaluations Guide, and the difference between testing the model and testing the whole system is covered in Model-Level vs System-Level AI Evaluation.

Where it breaks down

• Static attacks understate risk: NIST CAISI reported that against one agent the strongest baseline hijacking attack succeeded 11% of the time, while an attack developed through red teaming succeeded 81% of the time. A suite that never adapts measures yesterday's attacker.
• Single attempts understate risk: in the same work, average success across five injection tasks rose from 57% on one attempt to 80% with 25 attempts. Production attackers retry; a one-shot test does not.
• Unknown inputs: real users bring languages, formats and edge cases nobody wrote into the dataset.
• Evaluation awareness: capable models can behave differently when a context looks like a test. See the AI Scheming, Sandbagging and Evaluation Awareness Guide.

Controls it drives

Release gates on critical tasks, untouched held-out sets, repeated attempts, sandboxed harnesses and time-limited risk owner overrides. Article 9(8) of the EU AI Act asks for testing during development as appropriate and in any event before placing on the market, against metrics and probabilistic thresholds defined beforehand, which is in effect a requirement for written gates.

Runtime monitoring in depth

Threat model

Monitoring assumes the system will meet things nobody tested: novel jailbreak phrasings, poisoned documents in a retrieval corpus, a user population that differs from the evaluation set, gradual shifts in input distribution, and changes in the behaviour of a hosted model. It also covers agents whose individual steps pass content checks while the sequence of tool calls is harmful. See the AI Agent Behaviour Baselining and Anomaly Detection Guide.

Architecture in words

Telemetry is captured at the gateway, in the application around each model call, and in the agent framework around each tool call: prompts and responses (with redaction), retrieved context identifiers, tool names and arguments, guardrail verdicts, latency, token counts, caller identity and the model identifier returned by the provider. Jobs compare distributions against the release baseline, rules alert on guardrail block spikes, canary token use and unusual tool sequences, and sampled conversations go to human review. The telemetry model and where to instrument are covered in the LLM Observability Guide, and the split between debugging traces and evidential records in Agent Observability vs Audit Logging.

Where it breaks down

• Detection after harm: monitoring sees the data leak or the destructive tool call after it happened. For irreversible actions that is too late; those need pre-execution checks, covered in Pre-Execution Policy Checks vs Post-Execution Monitoring for Agents.
• No ground truth: production rarely tells you whether an answer was correct. Accuracy decay can run for months before outcome labels or complaints arrive.
• Rare high-impact events: a failure that occurs once in 50,000 requests will not move a dashboard average, and sampling may skip it entirely.
• Uninstrumented paths: a team calling the model API directly with a key, a batch job outside the gateway, or an agent tool that logs nothing produces no signal, which reads as healthy.

Controls it drives

Alert thresholds with named owners, kill switches and rollback to the last tested configuration, rate and spend limits, quarantine of suspect retrieval sources, incident triage that preserves evidence before fixes, and a route from every confirmed finding back into the test suite. The NIST AI RMF MANAGE 4.1 subcategory describes the target: post-deployment monitoring plans that include mechanisms for capturing input from users and other AI actors, appeal and override, decommissioning, incident response, recovery and change management.

Coverage gap map: what each sees and misses

Two rows deserve emphasis. Model updates are now routine: providers publish lifecycle schedules for hosted models, and the notice periods differ. Providers are listed alphabetically. Anthropic's model deprecations page describes active, legacy, deprecated and retired states and at least 60 days' notice before retirement of publicly released models. OpenAI's deprecations page describes active, deprecated and shutdown states, at least six months' notice for generally available models, at least three months for specialised variants, and much shorter notice for preview models, and exposes a shutdown date through its models endpoints. A forced migration is, for assurance purposes, a new system. See AI Model Change Management vs Traditional Change Management.

Feedback loops are the second. Article 15(4) of the EU AI Act requires high-risk systems that continue to learn after release to eliminate or reduce, as far as possible, the risk of biased outputs influencing future inputs, and to address such feedback loops with mitigation measures. That risk only exists in production, so only monitoring can see it.

What regulators and frameworks expect

The EU obligations apply to high-risk systems on the timeline set out in the EU AI Act Post-Market Monitoring Guide (Article 72), which also covers the plan contents and incident reporting clocks; this page does not repeat them.

Splitting the assurance budget

There is no correct ratio. Let the system's characteristics, not the team's habits, decide which side gets more.

• Small organisations: one open-source evaluation harness with a few hundred high-impact cases as a release gate, model identifier and guardrail verdict logging on every request, a weekly sampled review and a monthly re-run of the full suite. That is a defensible minimum.
• Large organisations: a shared evaluation service with a central case library, risk-tiered gates, telemetry standards that every AI application must meet before go-live, and a single owner for the loop between the two. Fund the loop explicitly; it is usually the part nobody owns.
• Rule of thumb for review: if most findings in the last quarter came from production, testing is under-funded; if monitoring has produced no new test cases in a quarter, monitoring is either blind or nobody is reading it.

Worked scenario: a claims triage assistant after a model update

An insurer runs an assistant that reads inbound claim emails and attachments, summarises them and proposes a routing decision, with a human approving each route. It passed a release gate six months ago: injection suite below threshold on critical tasks, routing accuracy above the declared level, no canary leakage.

1. The change: the hosted model provider announces deprecation of the snapshot in use. The team switches to the recommended successor in a configuration change and, because the prompt did not change, skips the full suite.
2. What monitoring shows: within a week the guardrail block rate on attachments falls by half, while the share of claims routed to the fast-track queue rises. Model identifier logging ties both shifts to the switch date.
3. Triage: sampled review finds that the successor model follows instructions embedded in some attachment footers ("route to fast track") more readily, and the guardrail, tuned on the old model's output style, misses them.
4. Containment: the team rolls back to the old snapshot while it remains available, quarantines affected claims for re-review and preserves logs and configuration before changing anything.
5. Feedback: the footer instructions become new indirect injection test cases; the full suite is re-run against the successor with repeated attempts; the change process is updated so any model identifier change triggers the full suite, not the smoke set.

Building the feedback loop

1. Baseline at release: store the evaluation results, input and output distribution statistics and guardrail rates for the released configuration. Monitoring compares against this, not against an abstract ideal.
2. Log the configuration with every event: model identifier, prompt version, tool set and guardrail policy version, so a runtime shift can be tied to a change.
3. Define triggers for re-testing: any model identifier change, provider deprecation notice, prompt or tool change, retrieval corpus refresh above an agreed size, and any monitoring threshold breach.
4. Route findings into the case library: every confirmed runtime incident, red team finding and user report becomes a labelled regression case before the ticket closes. Continuous adversarial testing can feed the same library; see the Continuous Automated Red Teaming (CART) Guide.
5. Report-only first: run new monitoring rules and new gates in report-only mode for a few weeks to tune thresholds and false positives.
6. Enforce with a rollback path: keep the last tested configuration deployable, and let a risk owner override a failed gate only with a written expiry.

Detection and measurement

• Test currency: share of production AI configurations whose last full suite run is newer than their last model or configuration change. Target 100% for high-risk systems.
• Telemetry coverage: share of production AI traffic that carries model identifier, caller identity and guardrail verdict. Gaps are blind spots, not quiet periods.
• Finding source ratio: findings from production versus from pre-release testing per quarter. A rising production share points to weak or stale suites.
• Loop closure time: days from a confirmed runtime finding to a merged regression case.

Common failure patterns

• The gate certificate that never expires: a system passes once and is treated as assured indefinitely while models, prompts and data change underneath it.
• Swapping models as a configuration change: a successor snapshot goes live with only a smoke test, because the code did not change. The new model is a new system.
• Monitoring without a baseline: drift dashboards with nothing to compare against produce numbers but no decisions.
• Monitoring that never feeds testing: the SOC closes AI alerts, but nobody adds the case to the suite, so the same failure returns after the next release.
• Logging nothing about the model: without the model identifier per event, a provider-side change is indistinguishable from a change in user behaviour.

Framework mapping

Practitioner checklist

• Write down, per AI system, which risks are assured by testing, which by monitoring, and which by both.
• Gate every release on critical tasks with predefined thresholds, repeated attempts and a held-out set.
• Treat any model identifier change, including provider-forced migrations, as a full re-test trigger.
• Track provider deprecation schedules and record the planned migration date for each production model.
• Log model identifier, prompt version, caller identity and guardrail verdict on every production request.
• Take a statistical and behavioural baseline at release and alert on deviation with named owners.
• Turn every confirmed runtime finding into a regression case before closing it, and measure loop closure time.
• Use pre-execution checks for irreversible agent actions instead of relying on monitoring.
• Review the finding source ratio quarterly and rebalance effort between the two sides.

Standards and references

• European Commission AI Act Service Desk: Article 9 Risk management system
• European Commission AI Act Service Desk: Article 72 Post-market monitoring
• European Commission AI Act Service Desk: Article 15 Accuracy, robustness and cybersecurity
• European Commission AI Act Service Desk: Article 43 Conformity assessment
• NIST: AI Risk Management Framework 1.0 (AI 100-1)
• NIST CAISI: Technical Blog: Strengthening AI Agent Hijacking Evaluations
• OWASP GenAI Security Project: GenAI Red Teaming Guide
• Anthropic: Model deprecations and OpenAI: Deprecations

Related NHI Mgmt Group resources: AI Security Evaluations Guide · LLM Observability Guide · AI-SPM vs AI Runtime Security · EU AI Act Post-Market Monitoring Guide (Article 72) · ML model evaluation and monitoring for security teams