AI vulnerability exploitation models are cheaper, but still miss depth

At a glance

What this is: This benchmark compares frontier AI models on vulnerability exploitation and finds that GLM-5.2 is the cheapest configuration tested, but not the most accurate.

Why it matters: It matters because security teams evaluating AI-assisted offensive testing, red teaming, and vulnerability triage need to understand when lower cost improves scale and when it still leaves exploitable gaps.

By the numbers:

• GLM-5.2 exploits 67% of the vulnerabilities at $1.13 per lead, the cheapest configuration we tested.

👉 Read TENZAI's benchmark analysis of GLM-5.2, GPT-5.5, and Opus-4.8

Context

AI vulnerability testing models are being judged less on novelty and more on whether they can find and exploit real weaknesses at a defensible cost. That changes the conversation from benchmark hype to operational trade-offs: coverage, reasoning depth, and repeatability matter more than raw model name.

For security programmes, the identity angle is indirect but real. When AI systems are used to probe applications, the resulting findings often map to access-control failures, secrets exposure, privilege paths, and other identity-adjacent weaknesses that determine how far an attacker can move once a flaw is found.

Key questions

Q: How should security teams evaluate AI claims in cybersecurity tools?

A: They should evaluate the tool by its actual decision behaviour, not by marketing language. Ask whether it learns from data, how it handles false positives, where humans intervene, and what evidence exists for performance in real environments. If the answer stays vague, treat the AI claim as unverified.

Q: Why does cost matter in AI-assisted vulnerability exploitation?

A: Cost determines whether a model can be used continuously or only in limited bursts. A cheaper model that finds fewer issues may still be valuable if it enables broader coverage, while a more accurate but expensive model may be reserved for complex cases. Programme value depends on matching spend to risk reduction.

Q: What should teams get wrong about AI exploitation benchmarks?

A: They often treat a benchmark score as proof of real-world readiness. In practice, exploitability depends on application type, vulnerability difficulty, available clues, and the model’s ability to chain actions into a working proof of concept. Benchmarks are useful, but only when they reflect the environment you actually need to defend.

Q: How do identity controls change when AI tools can generate exploit paths?

A: Identity controls become the blast-radius limiters after a vulnerability is found. Even when AI uncovers a weak point, least privilege, secrets hygiene, and privileged session controls determine whether the issue becomes a local defect or an incident with lateral movement and credential exposure. That makes IAM and PAM part of exploit containment.

Technical breakdown

How AI exploitation benchmarks measure recall and cost

The article uses recall as the primary metric, meaning the fraction of planted vulnerabilities an agent can actually exploit. That is different from simple detection or classification because the model must infer the bug from sparse clues, create a working proof of concept, and complete the exploit chain. Cost per lead then shows whether a model can do that work efficiently enough to be useful at scale. In practice, this is closer to offensive validation than static vulnerability scoring, because success depends on reasoning, tool use, and adaptation against varied enterprise applications.

Practical implication: teams should evaluate AI offensive tools on exploit success and cost per finding, not on benchmark labels alone.

Why higher reasoning effort changes exploitation outcomes

Reasoning effort acts like a compute and deliberation budget. At lower effort, a model may miss the right path even when it has enough information to solve the problem, while higher effort can improve chaining, hypothesis testing, and script generation. The key point is that improved results are not free. If a model only wins when you pay materially more, then its value depends on whether you need broad coverage, deep validation, or both. For defenders, this means model selection should match the task, not the hype cycle.

Practical implication: set different model tiers for broad scanning, deeper validation, and high-confidence exploit confirmation.

Specialised subagents make AI offensive tooling more operational

The benchmark description shows a fleet model where one subagent identifies a lead and another, Bonzai, handles exploitation and proof-of-concept creation. That division of labour matters because it turns a general model into a workflow with narrower, more measurable responsibilities. This is the same design pattern now emerging in agentic AI more broadly: orchestration plus task-specific execution. The governance lesson is that the risk is not just model capability, but how fast specialised subagents can translate a weak signal into an actionable exploit path.

Practical implication: inventory AI workflows by subagent role so you can control exploit generation, execution, and escalation separately.

NHI Mgmt Group analysis

Cost efficiency is becoming a governance issue, not just a procurement one. A model that can exploit 67% of planted vulnerabilities at low cost changes how teams scale offensive validation, because cheaper runs mean more frequent testing and wider coverage. That helps security programmes, but it also lowers the barrier to misuse by less sophisticated actors. The practical conclusion is that cost per exploit will become part of risk assessment, not just a line item in tooling selection.

AI offensive tooling is moving from single-model evaluation to workflow evaluation. The article’s subagent structure matters because exploitation is no longer a one-step model response, but a chained process that includes lead generation, reasoning, proof creation, and output packaging. That shifts governance toward orchestration controls, not just model benchmarking. Practitioners should evaluate the whole attack workflow, not only the base model.

Machine-executed vulnerability discovery increases pressure on identity and access controls downstream. Once an AI system can reliably identify exploitable paths, the most valuable weaknesses are often the ones that lead to privilege escalation, secrets exposure, or lateral movement. That is where NHIs and privileged access become the real blast-radius limiters. The result is a stronger need to treat AI testing outputs as inputs to identity governance, not isolated bug reports.

There is a new gap between exploitability and remediation readiness. Fast, low-cost discovery does not guarantee that teams can prioritise or fix what matters first. If AI tools can generate more valid findings than humans can triage, then vulnerability management becomes a sorting problem with identity and privilege consequences. The conclusion is that remediation pipelines must be able to distinguish cosmetic exposure from paths that actually expand attacker reach.

AI agent workflows need named governance concepts because generic labels are too vague for decision-making. Exploitation orchestration drift: when specialised subagents create a faster, less visible path from hypothesis to proof of concept, oversight breaks down unless teams define ownership, logging, and approval boundaries for each stage. Practitioners should govern the chain, not the model alone.

What this signals

AI-assisted offensive testing will increasingly stress identity governance because the highest-value findings are those that lead into credentials, privilege, and lateral movement. That makes IAM and PAM the downstream control plane for AI-discovered vulnerabilities, not just the response team after the fact.

Exploitability-to-remediation gap: security teams will need to classify findings by attacker value, not by discovery volume, because cheaper AI testing can create more valid outputs than human triage can absorb. The operational question becomes which findings expand identity reach, not which ones arrive first.

For teams aligning to broader control frameworks, the relevant standard lens is how quickly access paths can be constrained after validation. The NIST Cybersecurity Framework and related access-control practices remain the right reference points when AI tools shorten the time between vulnerability hypothesis and working exploit.

For practitioners

• Benchmark AI offensive tools against exploit success and cost Measure models on the percentage of planted vulnerabilities they can actually exploit, alongside cost per lead, so procurement choices reflect operational value rather than headline capability.
• Separate subagent responsibilities in AI security workflows Require different controls for lead generation, proof-of-concept creation, and execution so a single workflow cannot silently move from analysis to active exploitation.
• Prioritise findings that change privilege boundaries Triage AI-discovered issues by whether they expose credentials, enable privilege escalation, or open lateral movement paths, using access impact as the first filter.
• Track model reasoning tiers in offensive testing pipelines Use lower-cost models for broad discovery and reserve higher-reasoning configurations for complex exploits where deeper chain construction is necessary.
• Connect exploit findings to identity controls immediately Feed any successful exploit path into IAM, PAM, and NHI review queues so secrets, service accounts, and privileged sessions are reassessed before the next testing cycle.

Key takeaways

• AI vulnerability benchmarks are shifting from curiosity to operational decision support, with cost per exploit now a meaningful programme metric.
• A cheaper model that finds most vulnerabilities still leaves a governance gap if it cannot match the best models on deep exploit success.
• Identity and privileged access controls become more important after AI discovers a weakness, because they determine whether the exploit can spread.

Key terms

• Exploit Recall: Exploit recall is the fraction of planted or real vulnerabilities that a testing system successfully turns into working exploitation. It is a stronger measure than detection alone because it reflects end-to-end offensive capability, not just identification of a possible issue.
• Reasoning Effort: Reasoning effort is the amount of compute and deliberation a model uses before producing an answer or action. Higher effort can improve complex chain construction and tool use, but it also increases cost, which matters when testing needs to run repeatedly at scale.
• Agent Orchestration: Agent orchestration is the coordination of multiple AI agents or workflows to complete a task set with limited human intervention. In identity terms, it creates delegated execution paths that need ownership, scope limits, and auditability because work is no longer performed only by a person in one session.
• Exploitability Benchmark: An exploitability benchmark tests whether a model can progress from a weak signal to a working proof of concept against planted or known vulnerabilities. It is closer to real offensive validation than generic model scoring because it measures practical attacker outcomes.

What's in the full report

TENZAI's full research covers the benchmark mechanics and model-by-model comparisons this post intentionally leaves at summary level:

• Per-model recall curves across GLM-5.2, GPT-5.5, and Opus-4.8 at different reasoning levels
• Cost-per-vulnerability comparisons that show where higher reasoning effort changes the trade-off
• Benchmark setup details for the planted vulnerabilities and enterprise application scenarios
• The role of Bonzai and the subagent workflow in turning leads into exploit proof-of-concepts

👉 TENZAI's full research includes the model-by-model recall data and exploitation cost curves.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and machine identity security. It helps practitioners connect exploit paths to access control decisions across identity programmes.