AI security regression testing: repeatable tests for prompts, guardrails and agents

An AI application that passed its security review last month can fail it today without a single line of code changing, because the model, the system prompt, the retrieval corpus or a guardrail setting moved underneath it. Regression testing is the discipline that catches this: a fixed, versioned set of security test cases that runs on every change and on a schedule, and fails loudly when a previously closed weakness comes back. This guide goes deeper than the AI Security Evaluations Guide, which covers what to evaluate and how to design a suite; here the focus is the mechanics of keeping a regression suite trustworthy over time: change triggers, baselines, handling non-deterministic outputs, flaky tests, triage and maintenance. It is written for AI platform engineers, application security teams and the people who own release pipelines for prompts and agents. Sources were checked on 8 November 2026.

Key takeaways

• Treat prompts, guardrail configs and model versions as code. Every one of them can reopen a closed finding, so every one of them must trigger the regression suite.
• A regression is a change in rate, not a single failure. LLM outputs are probabilistic, so run each security case several times and compare attack success rates against a stored baseline.
• Score on what the system did, not what it said. Tool calls, outbound requests and canary hits are stable signals; free-text grading drifts and needs auditing.
• Every closed finding becomes a test before the ticket closes. Red team results, bug bounty reports and incidents feed the suite, which is how it stays relevant.
• Flaky tests destroy gates. If engineers learn to rerun until green, the suite stops protecting anything; quarantine, fix and track flakiness as a metric.

What makes AI regression testing different

Classic regression testing assumes the same input gives the same output, and that the system only changes when someone commits code. Neither holds for LLM applications and agents. Three differences shape everything else on this page.

• Non-determinism: sampling means one run of a jailbreak or injection case can pass and the next fail. A single green run proves very little, which is why frameworks build in repetition. The UK AI Security Institute's Inspect framework, for example, lets a task run several epochs and reduce the results with reducers such as mean, majority, max, pass_at_{k} and at_least_{k}.
• Changes you did not make: hosted models are updated by providers, retrieval indexes change as documents are added, and tool APIs evolve. The system under test moves even when your repository is quiet.
• Fuzzy pass criteria: "did not leak the system prompt" or "refused the harmful request" is harder to assert than an HTTP status code. Assertions range from exact string checks to model-graded rubrics, and each has a different failure mode.

OWASP's guidance for LLM01:2025 Prompt Injection lists adversarial testing and attack simulation, "treating the model as an untrusted user", as a standing mitigation alongside least privilege and human approval for high-risk actions. Regression testing makes that continuous.

What triggers a regression run

Map every change type that can affect security behaviour to a suite and a gate. Anything not on this list is a blind spot.

If the application calls a model alias the provider can move, record the resolved model identifier in every result so a regression can be tied to its cause. See the AI Guardrails and AI Firewalls Guide for the guardrail layer itself.

Building the regression case library

A regression suite is only as good as its cases. Each case should be a small, self-describing record rather than a loose prompt in a spreadsheet.

• Identifier and origin: a stable ID and where the case came from (red team report, bounty submission, incident, public technique, synthetic variant). Origin tells you who to ask when the case fails.
• Threat label: map each case to a framework entry such as OWASP LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM06:2025 Excessive Agency or LLM07:2025 System Prompt Leakage, and to MITRE ATLAS techniques such as AML.T0051 LLM Prompt Injection or AML.T0054 LLM Jailbreak. Labels make coverage measurable.
• Severity tier: critical cases (data exfiltration, destructive tool calls, credential use) gate releases; lower tiers inform trends.
• Input and environment: the user turn, any planted content (document, email, tool result) and the seeded state the agent will see.
• Oracle: the precise condition that counts as failure, ideally a state check.
• Paraphrase family: several rewordings of the same attack, so a fix that only blocks the exact string is caught.

Keep a held-out set that nobody uses while tuning prompts or guardrails. If the main suite passes and the held-out set does not, the team has fitted the defences to the tests. Open-source scanners help seed the library: garak runs probes against a model and applies detectors to the outputs, promptfoo can generate red team cases from plugins and strategies, and PyRIT runs scripted single and multi-turn attacks with scorers. See the garak Guide, the Promptfoo Guide and the PyRIT Guide; generated cases still need review before they become gating tests.

Oracles and assertions that stay stable

The oracle decides whether a regression suite produces signal or noise. Prefer the most deterministic check that captures the risk.

Tooling reflects this range. promptfoo, for example, offers deterministic assertions such as contains, regex, is-json, is-refusal and trajectory checks such as trajectory:tool-used and trajectory:tool-args-match, model-graded types such as llm-rubric and moderation, and a not- prefix to invert any assertion. Weights and thresholds let a test case pass on a weighted score, but for security cases a single forbidden action should fail the case outright; do not let a high score on harmless assertions average away a leak.

Baselines and handling non-determinism

Because one run proves little, define a regression in statistical terms and store the baseline it is measured against.

1. Fix the sampling settings for tests: record temperature and other generation parameters with each result. Testing only at temperature zero hides behaviour users will see in production if production samples at a higher setting.
2. Repeat each case: run critical cases several times (Inspect calls these epochs). Report both the mean attack success rate and "succeeded at least once in k attempts", because a retrying attacker only needs one success.
3. Store a baseline per case: keep the last accepted run's rates, together with the model identifier, prompt version, guardrail version, dataset version and scorer version. Without the versions a trend line is meaningless.
4. Set tolerances by tier: for critical cases the tolerance is usually zero, any success fails the gate. For lower tiers, flag a rate increase beyond an agreed margin rather than failing on noise.
5. Re-baseline deliberately: accepting a new baseline is a security decision. Record who approved it and why, the same way you would record a risk acceptance.

Run the false refusal set alongside the attack set. A guardrail change that blocks every injection but also blocks a fifth of legitimate requests is a regression too, and teams under pressure will loosen the guardrail in ways that are not tested.

Wiring it into the pipeline

Regression testing for AI uses the same pipeline patterns as other test stages, with a few additions.

• Pull request stage: a smoke set of critical cases on every prompt, tool or guardrail change. Keep it fast enough that developers do not route around it.
• Release candidate stage: the full library with repeated attempts and the held-out set.
• Scheduled stage: daily or weekly runs against a production mirror catch provider-side drift. promptfoo's documented GitHub Actions example, for instance, runs a red team scan on a daily cron schedule.
• Results as artefacts: publish machine-readable results (JUnit XML is supported by most CI viewers and by promptfoo through -o results.junit.xml) and keep full transcripts or logs for triage. Inspect writes logs that can be reopened in its log viewer and rescored later with inspect score, which lets you apply a corrected scorer without paying for new model calls.
• Secrets in the pipeline: the suite needs provider API keys. Use a dedicated, scoped key per environment from the pipeline's secrets store, never a developer's personal key. See the CI/CD Pipeline Identity Security Guide.

Test environments must not become the incident. Agents under test should get mock tools, canary data and fake credentials, with network egress denied by default. The AI Agent Sandboxing and Code Execution Guide covers the harness controls.

Triage, flakiness and suite maintenance

When a regression case fails, the question is whether the system regressed, the test is wrong, or the result is noise. A short, written triage path keeps gates credible.

1. Confirm with reruns: rerun the failing case with more attempts. A critical case that succeeds even once is a real finding, whatever the rerun shows.
2. Diff the versions: compare model identifier, prompt, guardrail, tool definitions and dataset against the baseline record to find the change.
3. Check the oracle: read the transcript. If a grader model flagged a safe answer, fix the scorer and rescore; do not delete the case.
4. Quarantine, do not delete: a flaky case moves to a quarantine list with an owner and a due date. It still runs and reports, but does not gate. Quarantine size is a metric.
5. Close the loop: a confirmed regression gets a ticket linked to the case ID, and the case gates the fix.

Each quarter, add cases from new public techniques, remove duplicates, refresh paraphrases and check coverage against the tools and data the agent can now reach.

Measuring whether it works

• Critical case pass rate at release: should be 100%, with any exception tied to a recorded risk acceptance.
• Coverage: share of tools, data connectors and applicable OWASP LLM risks with at least one gating case.
• Finding-to-case conversion: share of closed red team, bounty and incident findings that have a regression case. Aim for all of them.
• Escaped regressions: security issues found in production that an existing case should have caught. Each one is a suite defect.
• Flaky rate and quarantine age: share of cases with inconsistent results and how long quarantined cases stay unfixed.
• Time to detection after provider change: days between a provider model update and the scheduled run that tested it.

Where identity fits

Agents act with credentials, so identity abuse belongs in the regression library. Seed test environments with canary credentials and fake tokens that alert when used, and treat any use as a failure. Include cases where planted content asks the agent to reveal, reuse or forward its own tokens, request broader scopes or create accounts. When an agent's permissions change, the regression run is the check that the new scope cannot be reached through injection. The AI Agent Honeytokens and Canaries guide covers seeding these safely.

Framework mapping

Common failure patterns

• Prompt edits outside the pipeline: system prompts changed in a console or feature flag tool, never triggering the suite. Store prompts in version control or make the prompt store call the pipeline.
• Single-run gates: one pass per case declared green, while the same attack succeeds on the third try in production.
• Unpinned models with no scheduled runs: the provider updates the model, nobody tests, and a closed injection path reopens.
• Averaging away the critical case: a 97% overall pass rate hides the one case where customer data leaves the system.
• Rerun until green: engineers retry flaky gates until they pass, and the suite quietly stops gating anything.
• Findings closed without cases: a red team issue is fixed, the fix is later reverted, and there is no test to notice.

Practitioner checklist

• Put system prompts, guardrail configuration, tool definitions and model identifiers under version control and make each change trigger the suite.
• Give every case an ID, origin, threat label, severity tier, environment and state-based oracle where possible.
• Run critical cases several times and fail the gate on any success; track rates for lower tiers against a stored baseline.
• Record model, prompt, guardrail, dataset and scorer versions with every result.
• Keep a held-out set and a false refusal set, and run both before every release.
• Schedule runs against a production mirror to catch provider-side model changes.
• Quarantine flaky cases with an owner and a due date; never delete them to unblock a build.
• Convert every closed red team, bounty and incident finding into a gating case before the ticket closes.
• Use scoped pipeline API keys and sandboxed, egress-denied test environments with canary credentials.

Standards and references

• OWASP GenAI Security Project: LLM01:2025 Prompt Injection
• OWASP: Top 10 for LLM Applications 2025
• NIST: AI Risk Management Framework
• MITRE ATLAS: LLM Prompt Injection
• UK AI Security Institute: Inspect, Scoring Metrics and Reducing Epochs
• promptfoo: Assertions and Metrics
• promptfoo: CI/CD Integration
• garak: Reference Documentation

Related NHI Mgmt Group resources: AI Security Evaluations Guide · AI Red Teaming Guide · AI Agent Security Testing Checklist · Prompt Injection Guide · AI production acceptance criteria