CyberMetric shows cybersecurity knowledge is near saturation in LLMs

At a glance

What this is: This benchmark evaluates 13 models on 10,180 multiple-choice cybersecurity questions and finds tightly clustered scores that point to broadly internalized static knowledge.

Why it matters: For security and identity practitioners, the result matters because model selection for copilots, triage assistants, and policy support should shift from trivia recall to reasoning, tool use, and governance fit.

By the numbers:

• All 13 models scored between 86.9% and 90.8%, creating a gap of just 3.9 percentage points from top to bottom.
• GPT-5 Mini scored 25.45% without answer choices, which matches random guessing at 25%.
• Qwen3 Coder Next cost just $0.024 per 1, 000 questions while Claude Sonnet 4.5 cost $4.77, a 200x difference.
• 100% completion on the benchmark, mpletion on the benchmark, while GLM-4.7 finished at 97.4%.

👉 Read Cotool's CyberMetric benchmark analysis of cybersecurity model accuracy and cost

Context

CyberMetric is a multiple-choice cybersecurity benchmark, not an operational test of whether a model can defend systems, investigate incidents, or make trustworthy decisions under pressure. That distinction matters because many procurement conversations still overweight static knowledge scores when the real enterprise risk is whether a model can support security work safely, consistently, and with traceable outputs. In identity-heavy workflows, that includes how an assistant handles IAM policies, privilege questions, secrets handling, and incident context.

The benchmark’s tight score band suggests that, for broad cybersecurity facts, model differentiation is compressing. That does not remove risk. It shifts the decision problem toward the quality of reasoning, the reliability of tool use, and the governance controls around any model allowed to touch security data, identity systems, or privileged workflows.

Key questions

Q: How should security teams evaluate LLMs for cybersecurity use beyond benchmark scores?

A: Use benchmark scores as a baseline, then test the model in realistic workflows. Measure whether it can reason over incomplete evidence, use tools safely, respect access boundaries, and produce auditable outputs. A model that scores well on static questions may still fail in incident response, IAM analysis, or privileged operations if it cannot handle context and control intent.

Q: Why do high cybersecurity benchmark scores not guarantee safe enterprise use?

A: Because multiple-choice benchmarks mainly test stored knowledge, not judgment. Enterprise security work requires reasoning, state management, policy adherence, and accountability. A model can know the right answer to an access-control question and still make unsafe recommendations when the problem involves real identities, privileges, or secrets.

Q: What do security teams get wrong about model selection for SOC and IAM workflows?

A: They often optimise for leaderboard rank instead of workflow fit. For SOC and IAM use cases, the bigger questions are whether the model can be audited, whether it can be constrained by policy, and whether it fails safely when it encounters ambiguous or privileged requests.

Q: How should organisations decide which model can touch sensitive identity data?

A: Base the decision on sensitivity, privilege, and logging requirements. If a workflow involves identity records, privileged access, or secret handling, require stronger retrieval controls, narrower permissions, and explicit human review for actions that could alter access or exposure.

Technical breakdown

Why static cybersecurity benchmarks are saturating

CyberMetric is a closed-book, single-turn multiple-choice benchmark. That means the model only has to map stored knowledge to one of four answers, without retrieval, tools, or multi-step reasoning. Once a model has seen enough relevant material during training, gains become smaller because the task rewards recognition more than operational judgment. This makes benchmark scores useful for measuring baseline domain knowledge, but weak for distinguishing whether a model can handle messy security work, incomplete evidence, or context-sensitive access decisions.

Practical implication: treat static knowledge scores as a floor, not a buying decision.

Why cost and accuracy are no longer moving together

The benchmark shows a wide spread in cost for a very narrow spread in accuracy. That pattern is common in saturated evaluation tasks: larger or more expensive models can be only marginally better on simple recall, while cheaper models can still stay within a small accuracy band. For security teams, this means the economics of model selection depend less on raw benchmark ranking and more on where the model will sit in the workflow, what data it can access, and whether outputs are auditable.

Practical implication: price models by operational role, not by leaderboard position alone.

Why cybersecurity knowledge is not the same as cyber capability

A model can answer standards questions correctly and still fail at real security work. Operational capability requires reasoning across evidence, using tools, maintaining state, and respecting policy boundaries. In identity and access contexts, that matters because a model may know the definition of least privilege yet still mis-handle a privileged workflow, overstate confidence, or fail to preserve control intent when interacting with IAM, PAM, or secrets systems. Benchmarks that isolate memory do not measure those failure modes.

Practical implication: use applied benchmarks and workflow tests before letting a model influence security decisions.

NHI Mgmt Group analysis

Static benchmark saturation is changing what “good” means for security AI. When 13 models cluster within 3.9 percentage points on a 10,180-question test, leaderboard separation stops being a strong proxy for operational fitness. That does not make evaluation irrelevant. It means practitioners need to value tool use, traceability, and policy adherence over trivia recall. For identity teams, the same logic applies to assistants that advise on access, secrets, and privileged workflows.

CyberMetric exposes a verification trust gap in AI governance. The benchmark confirms that models can internalize cyber facts without proving they can apply them safely in live environments. That is the core governance problem for agentic systems and AI copilots: knowledge is easy to score, but safe action is much harder to validate. The practical conclusion is that access to identity systems should be gated by workflow tests, not by knowledge benchmarks alone.

Cost efficiency is becoming a governance variable, not just a procurement variable. A 200x price spread for a 2.4-point accuracy difference shows that evaluation strategy now affects architecture choices. Teams that tie every use case to the most expensive model are overpaying; teams that select purely on cost risk brittle or ungoverned behaviour. The right control is contextual model tiering tied to sensitivity, privilege, and auditability.

Benchmarking cybersecurity knowledge is not enough for NHI and agentic AI security. A model may answer access-control questions correctly while still mishandling delegated authority, secrets exposure, or runtime escalation. That gap matters because AI systems increasingly intersect with IAM and PAM, where a single bad action can matter more than a thousand correct quiz answers. Practitioners should test whether a model can operate within identity guardrails, not whether it can define them.

The named concept here is evaluation saturation. Once static knowledge tasks compress into a narrow score band, benchmark design has to move toward reasoning, tool interaction, and failure simulation. That is especially true for security domains where the cost of a wrong answer is operational, not academic. The field should treat evaluation saturation as a signal to redesign tests, not to assume the problem is solved.

What this signals

Model selection is moving from benchmark competition to control design. For security and identity programmes, that means asking whether an AI system can be constrained, observed, and revoked like any other privileged component, especially when it touches access decisions or sensitive operational context.

Evaluation saturation: once static cyber knowledge converges into a narrow band, teams need applied testing that simulates policy decisions, incident workflows, and identity actions. A model that knows the answer is not automatically one that can be trusted with the process.

The practical signal is to integrate model review into governance. Security leaders should expect procurement, risk, and IAM teams to assess auditability, tool permissions, and failure behaviour together, rather than treating model accuracy as the primary acceptance criterion.

For practitioners

• Separate recall from operational fitness Use static knowledge benchmarks only as one input. Add task-based testing for incident triage, policy interpretation, identity change requests, and tool-mediated workflows before approving a model for security use.
• Tier models by sensitivity and privilege Assign cheaper or smaller models to low-risk classification tasks, and reserve higher-trust workflows for systems with stronger audit logging, retrieval controls, and approval gates around IAM or secrets actions.
• Test identity workflows explicitly Run scenarios that force the model to reason about privileged access, service accounts, token handling, and access exceptions so you can see whether it respects least privilege in practice.
• Measure completion and failure modes together Do not stop at accuracy. Track refusal behaviour, malformed outputs, tool-call errors, and incomplete responses, because those failures are often the ones that break security operations.

Key takeaways

• CyberMetric shows that static cybersecurity knowledge is close to a saturation point for current LLMs.
• The meaningful differentiation now lies in reasoning, tool use, auditability, and failure handling, not in quiz-style recall alone.
• Security teams should govern models like privileged systems and test them against real workflows before granting access to sensitive identity or security data.

Key terms

• Static Cybersecurity Benchmark: A test that measures a model’s stored security knowledge using fixed questions and fixed answers. It is useful for baseline comparison, but it does not show whether the model can reason, use tools, or behave safely in a real security workflow.
• Evaluation Saturation: The point at which benchmark scores cluster so tightly that they stop revealing meaningful differences between systems. In security AI, saturation is a signal to move from recall tests to applied scenarios that measure decision quality, control adherence, and operational failure modes.
• Workflow Fit: The extent to which a model can perform safely inside a specific operational process, such as SOC triage or identity review. It includes auditability, refusal behaviour, policy compliance, and the ability to handle ambiguous inputs without creating control risk.
• Tool-Mediated Reasoning: A model’s ability to use external systems, data sources, and actions while keeping its outputs aligned with policy and context. This matters because many enterprise security tasks cannot be solved by knowledge alone and require controlled interaction with live systems.

What's in the full report

Cotool's full benchmark analysis covers the methodology and model-by-model detail this post intentionally leaves for the source:

• Per-model accuracy, cost, and completion comparisons across all 13 tested systems
• Method notes on the 10,180-question dataset and the human verification process
• Evidence behind the memorization check using answerless prompts and random-guess baselines
• Benchmarks that Cotool recommends for applied reasoning and agentic evaluation after static knowledge saturates

👉 Cotool's full benchmark post includes the per-model breakdown, methodology, and applied evaluation recommendations.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It helps security and identity practitioners turn governance principles into operating controls.