Most AI red teaming tools will produce a long list of "successful jailbreaks" against almost any model, which tells a buyer very little about whether their own application, agents and data are at risk. The market now spans open-source scanners and frameworks, commercial testing platforms and red teaming delivered as a service, and the same words ("continuous", "agentic", "compliance-ready") are used for very different things. What separates useful tools is attack coverage that matches your architecture, scoring you can trust, safe handling of the prompts and outputs it generates, and the ability to run the same tests in your pipeline after every change. This vendor-neutral buyer's guide is for AI security leads, application security teams and AI platform owners choosing a tool. It ends with a proof of concept you can run in three to four weeks.
Key takeaways
- Test the system, not just the model. Retrieval, tools, agents, memory and output handling create most real exposure; a tool that only sends prompts to a bare model endpoint misses them.
- Scoring is the product. Automated judges decide what counts as a failure. Ask for measured agreement with human reviewers on your data, and keep a human-labelled sample in every run.
- Coverage should map to named frameworks. Look for findings tagged to the OWASP Top 10 for LLM Applications (2025 and 2026 numbering), the OWASP Top 10 for Agentic Applications and MITRE ATLAS technique IDs.
- The tester is itself a risk. It holds API keys to your model endpoints, generates harmful content on purpose and may send data to a vendor's attacker model. Scope it like any privileged non-human identity.
- Open source and commercial are complements. Many teams run an open-source framework in CI and buy a platform or service for depth, reporting and expert-led exercises.
Define requirements
Start with an inventory of what you will test, then decide which delivery model fits. Write these into the RFP.
- Targets in scope: hosted model APIs, self-hosted models, retrieval-augmented generation (RAG) applications, chat interfaces behind authentication, agents with tools or MCP servers, and multimodal inputs (images, audio, documents).
- Who runs it: developers in CI, a central AI security team, or an external red team. This decides whether you need a CLI and config files, a web console, or a report.
- Obligations: internal release gates, customer assurance, the NIST AI RMF and its Generative AI Profile (NIST AI 600-1), whose action MS-2.7-007 calls for AI red-teaming against attacks such as prompt injection and data poisoning, and for providers of general-purpose AI models with systemic risk, Article 55(1)(a) of the EU AI Act, which requires "conducting and documenting adversarial testing of the model".
- Data constraints: whether test prompts, responses and retrieved documents may leave your environment, where they are stored and for how long. See the LLM Security Guide.
- Environments: whether testing will run only against staging copies or also against production, and who approves each campaign.
| Delivery model | Strengths | Limits | Best fit |
|---|---|---|---|
| Open-source frameworks and scanners (documented examples: PyRIT, garak, promptfoo) | Transparent probes and scoring logic; runs locally or in CI; no licence cost; extensible targets (garak supports REST endpoints, PyRIT supports HTTP, WebSocket and browser-driven targets) | You own maintenance, judge quality, triage and reporting; agent and multi-turn depth varies by project | Engineering-led teams, regression testing, organisations that cannot send data out |
| Commercial testing platforms | Curated and updated attack libraries, multi-turn and agentic attack generation, dashboards, framework-tagged reports, scheduling, team workflow | Attack and scoring logic often opaque; data may be processed in the vendor's cloud; pricing by target, scan or token | Several AI applications, central AI security function, continuous testing at scale |
| Red teaming as a service | Human creativity, business-logic and abuse-case testing, social and multi-step attacks, independent attestation | Point in time; quality depends on the people; findings need translating into repeatable tests | Pre-launch of high-risk systems, regulator or customer assurance, testing agents with real permissions |
Size changes the answer. A small team with one LLM feature can get far with an open-source framework in CI plus an annual external exercise. An enterprise with dozens of applications and agents usually needs a platform for scheduling and reporting, plus human red teamers for novel attack paths.
Evaluation criteria
Weight the criteria to your requirements. An illustrative split is 25% attack coverage, 20% scoring reliability, 15% agent and system testing, 15% safety and data handling, 10% CI and workflow, 15% reporting and everything else.
| Criterion | What good looks like | Weak signs |
|---|---|---|
| Attack library coverage | Findings mapped to OWASP LLM Top 10 IDs in both editions (for example LLM07:2025 System Prompt Leakage became LLM08:2026 Hidden Context Exposure; Excessive Agency moved from LLM06:2025 to LLM03:2026) and to MITRE ATLAS techniques such as AML.T0051 LLM Prompt Injection (.000 Direct, .001 Indirect), AML.T0054 LLM Jailbreak, AML.T0056 Extract LLM System Prompt and AML.T0057 LLM Data Leakage; documented update cadence | A fixed list of old jailbreak templates; no mapping, or mapping you cannot inspect |
| Agent and tool-use testing | Indirect injection through retrieved documents, web pages and tool outputs; tests for unauthorised tool calls (ATLAS AML.T0053 AI Agent Tool Invocation), broken object-level and function-level authorisation through the agent, and the OWASP Agentic Top 10 risks | Only tests the chat box; cannot observe or assert on tool calls |
| Multi-turn and multimodal | Adaptive multi-turn attacks (Crescendo-style escalation, tree or branching searches), encodings (Base64, ROT13, homoglyphs), and image, audio and document payloads | Single-turn only; multimodal "on the roadmap" |
| Scoring and evaluator reliability | Mix of deterministic checks and model-graded judges; configurable rubrics and examples; published or measured agreement with human labels; per-finding evidence (full transcript); repeat runs to show variance | A single attack success rate with no transcripts; judge model and prompt undisclosed |
| CI integration | CLI and API, config as code, pass or fail thresholds, regression replay of previously failed attacks, run tagging with commit and build IDs, results exportable as JSON | Console-only; no way to fail a build |
| Reporting for regulators and customers | Reports by framework (NIST AI RMF Measure, EU AI Act), scope, method, dates, tester independence, findings, retest status; evidence retained for audit | Glossy summary with no method or scope; no retest history |
| Safety of testing | Target allowlists, rate and cost limits, kill switch, dry-run mode, separate test accounts, no destructive tool calls without approval | Can be pointed at any URL; no spend or rate control |
| Data handling | Local or in-tenant attack generation option; clear statement of which prompts and outputs leave your environment; retention controls; no training on customer data | Remote generation on by default with no documented opt-out or data flow |
Testing production systems safely
Red teaming an agent means asking it to misbehave while it holds real permissions. Run against staging copies with synthetic data first, use dedicated test identities with the minimum permissions the agent needs, and set hard limits on tokens, spend and outbound destinations. Misconfigured evaluations have already reached real targets: see the Anthropic Claude evaluation incidents and the UK AISI agent testing incident. See the Red Teaming AI Agents for Identity Abuse.
The tool's own credentials
Every testing tool holds API keys or tokens to the model endpoints and applications it attacks. Open-source scanners commonly read them from environment variables (garak documents variables such as OPENAI_API_KEY; PyRIT ships .env example files), and commercial platforms store them in their own tenant. A leaked key lets an attacker run up your model bill or reach the same internal applications the tester can reach. Require per-target keys with spend limits, storage in your secrets manager or the vendor's encrypted vault with rotation, short-lived credentials where the provider supports them, and an inventory of every key the tool holds. See the LLMjacking Guide and the API Key Management Guide.
Questions to ask vendors
- Show a finding mapped to both the 2025 and 2026 OWASP LLM Top 10 IDs and to a MITRE ATLAS technique ID. How often is the attack library updated, and how do we see what changed?
- How does your judge decide an attack succeeded? Which model and rubric does it use, what is its measured agreement with human reviewers, and can we supply our own rubric and labelled examples?
- Run an indirect prompt injection planted in a document our RAG application retrieves, and show the tool call the agent attempted.
- Which attack generation runs in your cloud, which can run in our environment, and which of our prompts, outputs and retrieved documents leave our tenant? For how long are they kept?
- How do you stop a campaign from reaching systems outside the agreed scope, and what spend, rate and destination limits can we set?
- How are our model API keys and application credentials stored, scoped and rotated, and can we use our own secrets manager?
- Can the same tests run in CI with pass or fail thresholds, and can previously failed attacks be replayed automatically after a fix?
- What does a report for an EU AI Act Article 55 or NIST AI RMF evidence pack contain: scope, method, dates, tester independence and retest results?
- For a service engagement, who are the testers, how are they independent of the model's developers (NIST AI 600-1 action MS-1.3-003), and do we receive the attack prompts as reusable tests?
Red flags
- Headline attack success rates with no transcripts, no judge details and no repeat runs.
- Claims to certify a model or application as "safe" or "compliant" after one scan.
- No way to test tool calls, retrieval or multi-turn conversations.
- Prompts and outputs sent to a vendor-hosted attacker model by default, with no in-tenant option or data flow diagram.
- Requests for a single long-lived, unrestricted API key to your production model account.
- No scope controls, no kill switch and no rate or cost limits.
- Attack libraries made only of static jailbreak templates. OWASP's 2026 Prompt Injection entry advises testing against adaptive attackers who know the deployed defences and rejecting static-only attack-success claims.
Proof of concept
- Scope it: pick three targets, for example a public chatbot, an internal RAG assistant and one agent with tools. Agree weights and pass criteria before any demo.
- Build a known-answer set: plant 20 to 30 weaknesses you know about (a system prompt containing a canary string, a poisoned document with an indirect injection, a tool that should never be called) plus normal conversations. Measure how many each tool finds.
- Set up safely: run in staging, issue dedicated test credentials with spend limits and an expiry, restrict outbound destinations, and turn off any remote generation you have not approved.
- Check the judge: have two people label 100 transcripts and compare with the tool's verdicts. Record false positives, false negatives and run-to-run variance.
- Wire it into CI: run the suite on a pull request, fail the build on a threshold, fix one finding and confirm the regression replay passes.
- Produce the evidence pack: generate the framework report and check it states scope, method, dates and retest status.
- Roll back: revoke every test credential, confirm the tool can no longer reach the targets, and get written confirmation that trial data has been deleted.
Track a few numbers after go-live: share of AI applications and agents under scheduled testing, known-weakness detection rate, judge agreement with human labels, time from finding to fixed and retested, and the number of credentials the tool holds without an owner or expiry. See the Prompt Injection Guide and the AI Guardrails and AI Firewalls Guide for the controls your findings should feed.
How NHI Mgmt Group can help
We provide independent AI security and agentic AI identity advisory, requirements and evaluation support. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: Red Teaming AI Agents for Identity Abuse · LLM Security Guide · Prompt Injection Guide · AI Guardrails and AI Firewalls Guide · AI Security Platform Buyer's Guide