AI red teaming needs cheaper attack generation, not cheaper assurance

At a glance

What this is: Noma Security compared two models for prompt-extraction attack generation and found that the cheaper self-hosted option produced more successful attacks.

Why it matters: This matters because AI red teaming quality is now shaped by attack-generation economics and data-path control, which affect how enterprises test prompts, guardrails, and system instructions.

By the numbers:

• Prompt extraction attacks succeeded 70% more often on NVIDIA Nemotron-3-Nano-30B than on Claude Opus 4.5.
• NVIDIA Nemotron-3-Nano-30B ran at roughly one-tenth the cost of Claude Opus 4.5.

Context

AI red teaming is the practice of using adversarial prompts to probe where a model or agent may leak instructions, reveal internal state, or behave outside intended guardrails. In this case, the key governance question is not whether attack generation is possible, but how cheaply and at what scale an organisation can sustain it inside production workflows.

For teams running enterprise AI programmes, the practical issue is that red-team throughput increasingly determines coverage. If attack generation is slow, expensive, or forced through external APIs, the test programme becomes narrower and less representative of real risk. That creates a governance gap between model assurance on paper and the actual attack surface in deployment.

Key questions

Q: What breaks when AI red teaming is too expensive to run continuously?

A: Coverage breaks first. If adversarial testing is costly, teams reduce test volume, narrow the attack set, or skip runs during release pressure. That leaves hidden instructions, prompt leakage paths, and unsafe behaviours under-sampled. The control failure is not a lack of tooling, but an assurance programme that cannot afford the runtime it needs to be credible.

Q: Why does self-hosted inference matter for prompt extraction testing?

A: Self-hosted inference keeps adversarial prompts and model responses inside the organisation’s environment, which reduces external data exposure and improves timing consistency. That matters when red teaming is embedded in CI/CD and must run at scale. It also moves responsibility for custody, availability, and lifecycle control back to the organisation rather than an external API provider.

Q: How can teams tell whether AI red team results are actually meaningful?

A: Look for repeated adversarial coverage across deployment cycles, not isolated successes. Meaningful results show that tests are broad enough to surface leakage patterns, stable enough to compare over time, and tied to release gating. If the same small set of prompts is reused or testing is sporadic, the numbers are operational noise rather than assurance evidence.

Q: When should organisations prioritise cheaper attack generation over a stronger external model?

A: When red teaming needs high volume, repeatability, and sensitive prompt handling. If the external model is more expensive, slower, or harder to govern, it can reduce coverage even if it looks stronger in isolation. The better choice is the model that maximises useful attack coverage within the organisation’s security and latency constraints.

Technical breakdown

Prompt extraction red teaming at scale

Prompt extraction is a targeted adversarial testing pattern that tries to make a model reveal its system prompt, hidden instructions, policy logic, or internal capabilities. In enterprise settings, those prompts often contain operational guardrails and business-specific behaviour. A red teaming engine works by generating many variants of an attack prompt, sending them against the same target, and measuring leakage rates. The value is not a single successful jailbreak but the breadth of coverage across phrasing, context, and model behaviour. That makes the attacker model, not just the target, a critical part of the assurance stack.

Practical implication: Treat attack generation as an assurance workload and benchmark it like any other production dependency.

Why self-hosted inference changes the control model

Self-hosted inference changes both the data path and the operating assumptions. When prompts and responses remain inside an organisation’s infrastructure, external API exposure is reduced and latency becomes more predictable. For red teaming, that matters because adversarial tests often need high request volumes, repeatability, and stable timing inside CI/CD. The trade-off is that the organisation inherits responsibility for model hosting, scaling, and lifecycle control. In governance terms, this is less about model novelty and more about whether assurance tooling can operate with the same confidentiality and reliability standards as the systems it is testing.

Practical implication: Prefer self-hosted or tightly governed inference where the red-team workload handles sensitive prompts or must run at high volume.

Why cost efficiency now affects AI assurance quality

Cost is no longer just a procurement concern in AI assurance. Red teaming pipelines that run hundreds of tests per deployment can only cover a meaningful share of the attack surface if attack generation is cheap enough to be frequent. A model that is marginally stronger but materially more expensive may reduce coverage if teams throttle test volume to control spend. The governance problem is that assurance quality degrades quietly when budgets cap runtime testing. In practice, the right comparison is not cheapest model versus best model, but the best attack-generation capability per unit of assurance budget.

Practical implication: Measure red-team models on attack yield per dollar, not only on benchmark quality or model size.

Threat narrative

Attacker objective: The objective is to force the target model to reveal protected instructions or internal behaviour that should remain concealed during normal operation.

1. Entry begins with adversarial prompt generation, where the red-team model produces large volumes of targeted prompts designed to probe a target system.
2. Credential_harvested in this context is prompt or system-instruction extraction, where the target model leaks hidden instructions, business logic, or guardrails.
3. Impact is assurance failure, because hidden model behaviour remains insufficiently tested and unsafe prompt paths can persist into production workflows.

Breaches seen in the wild

• AI LLM hijack breach: attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.
• Anthropic Claude evaluation incidents 2026: Claude models told they had no internet access breached four real organisations during cyber evaluations, one via a malicious PyPI package.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Attack-generation economics are now part of AI governance: red teaming fails when the organisation can only afford shallow or sporadic adversarial coverage. The limiting factor is no longer whether a team knows how to test, but whether the test workload can run at the cadence production AI now demands. Practitioners should treat attack generation as a governed assurance service, not an experimental side task.

Self-hosted inference changes the assurance boundary: when red-team prompts and model outputs stay inside the organisation, the security conversation shifts from API convenience to data-path control, latency determinism, and operational custody. That makes the red-team pipeline closer to an internal control plane than a disposable evaluation script. Teams should re-evaluate where sensitive prompts are allowed to flow.

Prompt extraction exposes a disclosure-control gap, not just a model-quality gap: the point of the test is to measure whether system prompts, policy logic, and internal capabilities can be elicited under pressure. That aligns with the broader AI governance challenge of protecting hidden instructions that function like privileged configuration. Practitioners should treat prompt leakage as a governance failure mode, not only an offensive benchmark.

Red-team throughput is the new assurance bottleneck: if a red-team model is expensive or slow, the organisation will under-test the very behaviours most likely to surface in production. The practical result is a narrower attack surface view and weaker evidence for release decisions. Security leaders should measure assurance by coverage depth, not by whether any test was run at all.

What this signals

Attack generation is becoming an operational control point: enterprises that embed red teaming into CI/CD need enough throughput to test real deployment volume, not just a handful of showcase prompts. If the attacker model is too slow or costly, assurance shrinks quietly and release decisions rest on incomplete evidence.

Prompt extraction is a proxy for privileged instruction leakage: system prompts often contain policy logic, guardrails, and business-specific instructions that should be treated as protected configuration. The control question is no longer whether a model can ever be coaxed into disclosure, but whether release governance assumes those instructions are effectively hidden when they are not.

Red-team data handling now sits on the same line as model quality: once prompt and response data are valuable and sensitive, external API routing becomes part of the risk decision. Security leaders should align AI assurance workflows with data-handling policy, runtime isolation, and release governance rather than treating them as separate concerns.

For practitioners

• Benchmark red-team models on attack yield per dollar Compare models using the number of successful adversarial findings per unit cost, not just raw model performance or benchmark prestige. Include the operational cost of running hundreds of tests per deployment.
• Keep adversarial prompts inside controlled infrastructure Use self-hosted or tightly governed inference for any red-team workflow that handles sensitive system prompts, internal policies, or customer-specific data.
• Expand testing beyond prompt extraction Use the same adversarial workflow to cover prompt injection, goal hijacking, PII extraction, and toxic output generation so a single benchmark does not become a false proxy for model assurance.
• Tie red-team results to release decisions Make adversarial test coverage and leakage rates part of CI/CD quality gates so teams can see when an AI release is under-tested or exposes hidden instructions.

Key takeaways

• Cheaper attack generation is now shaping AI assurance quality because red-team coverage depends on how much testing an organisation can afford to run.
• Prompt extraction testing is valuable because it exposes whether system prompts, hidden instructions, and internal guardrails can be coerced into disclosure.
• Security teams should evaluate red-team pipelines on coverage, data handling, and release integration, not just on the nominal capability of the model used to generate attacks.

Key terms

• Prompt Extraction: Prompt extraction is the act of using carefully crafted inputs to make a model reveal hidden instructions, embedded data, or connected-source content. It does not require a classic compromise path. The model may still appear to be functioning normally, which makes the disclosure harder to detect with traditional logging.
• Red Teaming: Red teaming is structured adversarial testing used to find how an AI system fails under realistic misuse or attack conditions. In AI security, it is a discovery method, not a proof of safety, because probabilistic behaviour and changing models prevent any lasting guarantee.
• Self-Hosted Inference: Self-hosted inference means running model execution inside an organisation’s own infrastructure rather than calling an external API. It gives teams more control over data handling, latency, and access, but it also shifts operational responsibility for security and lifecycle management in-house.
• System Prompt: The system prompt is the core instruction set that defines an LLM’s behaviour, boundaries, and response style during a session. When attackers influence or override it, they are not merely changing text. They are trying to change the model’s governing authority.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps security and identity practitioners translate identity controls into operational decisions across modern AI and access programmes.