TL;DR: Grok 4.5 reaches roughly 93% solve rate in XBOW’s benchmark and, between about $1 and $10 of spend, outperforms the other models it tested for practical offensive security workloads, according to Xbow. The finding shows that middle-band pricing can matter as much as raw capability when security teams choose AI tools for repeated cyber tasks.
At a glance
What this is: Xbow says Grok 4.5 delivers near-frontier offensive security performance, with its strongest results appearing in the middle of the cost curve rather than at the cheapest or most expensive end.
Why it matters: For security and identity teams experimenting with AI-driven testing or cyber workflows, the finding changes procurement from a simple capability race into a question of repeatable cost, runtime fit, and task selection.
Context
AI security testing is no longer only about whether a model can solve a task. The harder governance question is whether it can solve enough of the right tasks, at a spend level that makes repeated use viable inside real programmes. In practice, that means the economics of model choice can shape how far teams push AI into offensive testing, validation, and workflow automation.
For identity and security programmes, this matters because AI tools increasingly sit inside controlled workflows rather than as one-off experiments. When a model becomes the default for repeated testing, selection criteria move from raw benchmark performance to operating cost, reliability, and fit with the work being automated. That is the real governance shift this article exposes.
Key questions
Q: When does an AI security testing model become too expensive to scale?
A: A model becomes too expensive when the cost per additional useful result starts rising faster than the value of the extra coverage. In security testing, that usually appears in iterative workflows where the first few passes are productive but later passes add little. The right test is the budget band your programme actually uses, not the model's peak benchmark score.
Q: Why do repeated cyber workloads change how teams should evaluate models?
A: Repeated workloads expose the full economics of a model, including cumulative spend, consistency across iterations, and how quickly returns diminish. A model that looks strong once may become impractical when used dozens or hundreds of times. Security teams should therefore evaluate repetition behaviour, not just single-shot capability, before operationalising an AI tool.
Q: What breaks when a security team chooses a model only on headline benchmark scores?
A: Headlines can hide the point at which a model becomes too costly or too uneven for operational use. A high score may be meaningless if the model needs far more iterations or spend than the workflow can tolerate. That leads to overpaying for capability the programme cannot realistically consume.
Q: Should teams use the same model for reconnaissance, exploit crafting, and validation?
A: Usually not. Those workloads often need different blends of speed, persistence, and cost efficiency, so one model rarely wins across all three. A better approach is to assign models by task class and measure whether each one sustains useful output at the budget level required for that stage of the workflow.
Technical breakdown
Why budget bands matter more than single benchmark scores
A benchmark score in isolation can hide where a model is actually useful. In operational security work, the first question is not whether a model can solve a problem eventually, but whether it can do so at the spend level and iteration count that the workflow allows. A model that peaks only after high cumulative cost may be less useful than one that delivers strong progress early. This is why model economics and model capability need to be assessed together, especially in agentic or repeated-testing environments.
Practical implication: Evaluate models against the budget band in which the workflow will actually run, not just against headline solve rates.
What repeated cyber workloads change about model selection
Repeated security tasks behave differently from one-off prompts. As soon as a model is used for iterative reconnaissance, validation, or exploit-oriented testing, runtime cost, consistency across runs, and diminishing returns become part of the control problem. A model that is acceptable in a single trial may become expensive or uneven when scaled across many cases. That changes procurement logic for cyber operations, because the relevant question becomes how the model performs across the full loop of repeated use, not just at the start of the task.
Practical implication: Test AI tools under realistic iteration patterns before making them part of a recurring security workflow.
Offensive security testing needs capability and economy together
Offensive security workloads are especially sensitive to the trade-off between speed, depth, and cost. A model that reaches strong results only at the high end of spend may be unsuitable for broad testing programmes, while a cheaper model may stall before finding enough of the attack surface. The practical sweet spot is a model that can sustain useful progress without forcing the programme into premium pricing. That is why middle-of-curve performance can matter more than top-end theoretical strength for day-to-day security testing.
Practical implication: Choose models that preserve useful output across the whole attack-testing session, not just at one end of the price curve.
Threat narrative
Attacker objective: The objective is to improve the speed and depth of security testing output while keeping per-run cost low enough for repeated use.
- Entry occurs through the AI-assisted testing workflow, where the model is used to probe systems, generate attack ideas, or validate weaknesses.
- Escalation comes from iterative reasoning and longer runs, which increase the model's ability to refine paths and deepen testing output.
- Impact is improved vulnerability discovery and more efficient offensive security coverage, which changes how much testing a team can perform for a given budget.
Breaches seen in the wild
- DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
- 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Middle-market economics are now a first-order governance variable in AI security testing. A model that is merely strong in isolation is not automatically the right operational choice. When teams run repeated cyber workloads, cost per useful result becomes part of security architecture, not just procurement. Practitioners should treat budget-band fit as a control decision, not a convenience metric.
Offensive security work exposes the gap between benchmark success and programme utility. The article shows that the best model depends on iteration count, spend tolerance, and the point at which diminishing returns begin. That means benchmark leaderboards alone are too blunt for security operations. Teams need to map model choice to task class, runtime length, and acceptable cost envelope.
AI security testing is becoming an optimisation problem for governance, not just capability. Once a model is used repeatedly, the real issue is whether it can sustain reliable performance at scale without distorting programme economics. That shifts decision-making toward control of spend, workload fit, and repeatability. Security leaders should expect model economics to sit alongside accuracy and safety in AI governance reviews.
Repeated-use AI creates its own form of operational dependency. If a testing programme begins to rely on one model for the middle of the work distribution, the organisation is implicitly accepting a performance profile and a cost curve. That makes model selection part of resilience planning, especially where validation, recon, and exploit simulation are tied to release or assurance cycles. Teams should govern model dependency as they would any other critical control point.
From our research library:
- Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
- Read next: AI Security Platform Buyer's Guide
What this signals
The practical lesson is that model choice is becoming a control-plane decision for AI-enabled security operations. Teams that treat every cyber workload as the same problem will overfit to a single benchmark and miss the cost dynamics that decide whether a model can be used repeatedly.
Budget-band governance: define the spend range, iteration limit, and task class before you approve an AI model for security testing. That gives leaders a way to compare runtime value instead of reacting to raw performance claims.
Identity and access teams should pay attention where AI testing tools move from experimentation into routine workflows, because repeatability creates dependency. Once a model becomes part of assurance or validation cycles, governance has to include cost control, workload segregation, and fallback options.
For practitioners
- Define budget-band test cases Measure each candidate model at the spend ranges where real security tasks will run, not only at idealised benchmark settings.
- Separate models by workload class Use different models for reconnaissance, exploit crafting, and validation when the cost-performance profile differs by task.
- Track solve rate against cumulative spend Report the point where each model stops delivering additional useful output for the money consumed.
- Validate repeated-run consistency Replay the same security workflow multiple times to see whether the model remains stable as iterations increase.
Key takeaways
- AI security testing now depends on where a model sits on the cost curve, not just on whether it can solve difficult tasks.
- Xbow's analysis indicates that mid-band pricing can deliver the best practical trade-off for repeated offensive security work.
- Security teams should evaluate AI models by workload class, repeated-run behaviour, and cumulative spend before operational adoption.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATLAS | Adversarial ML tactics and techniques | The article concerns AI model behaviour in offensive security testing workloads. |
| Recommendation — Map AI-enabled attack testing against adversarial ML tactics and assess iteration-driven abuse paths. | ||
| NIST AI RMF | MEASURE — AI Measurement and Evaluation | The piece is about evaluating model performance across cost and workload conditions. |
| Recommendation — Measure model utility by workload, cost band, and iteration count before operational approval. | ||
| NIST AI 600-1 | GenAI risk profile | The article is about practical GenAI use in cyber tasks and the operational risks of deployment choices. |
| Recommendation — Apply the GenAI profile to evaluate performance, cost, and misuse risk in repeated security workflows. | ||
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy | AI model selection here is a risk and economics decision tied to programme governance. |
| Recommendation — Incorporate AI workload cost and reliability into your risk management strategy for security tooling. | ||
| MITRE ATT&CK | TA0007;TA0006 — Discovery; Credential Access | The article discusses offensive security workloads that probe systems and credentials. |
| Recommendation — Use ATT&CK mapping to classify what the model is helping test and where coverage remains weak. | ||
Key terms
- Budget-band fit: The alignment between a model's cost profile and the spend range where a security workflow actually runs. A model can be technically strong yet operationally poor if it only performs well outside the budget that the programme can sustain.
- Repeated-run consistency: The degree to which a model produces stable, useful output across multiple passes of the same task. In security operations, this matters because iterative workflows expose variance that single-shot benchmarks can hide.
- Offensive security workload: A task class focused on probing systems for weaknesses, validating exploitability, or simulating attack paths. In AI-assisted testing, the workload type determines whether speed, depth, or cost is the dominant constraint.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners connect identity control to the security workflows that increasingly depend on AI-enabled automation.
Published by the NHIMG editorial team on August 11, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org