Topic starter
18/08/2026 2:47 pm
TL;DR: A benchmark of frontier models on realistic SecOps tasks found GPT-5 reached 62.7% accuracy, Claude Haiku-4.5 was the fastest at about 240 seconds, and GPT-5 variants occupied the strongest performance-cost frontier, according to Cotool. The results suggest enterprise teams need to treat model selection as an operational control, not a generic AI choice.
Editorial analysis by NHI Mgmt Group, based on content published by Cotool: “Evaluating AI Agents in Security Operations”.
Key questions
Q: How should security teams evaluate SecOps agents before putting them near production data?
A: Use task-level evals that mirror real investigations, then measure accuracy, completion rate, latency, and tool behaviour together.
Q: Why does a high benchmark score not guarantee safe AI use in SecOps?
A: Because SecOps value depends on more than getting the final answer right.
Q: What do security teams get wrong about AI-assisted investigations?
A: They assume the model is the main value.
Practitioner guidance
- Define workflow-specific acceptance thresholds Set separate pass criteria for investigation accuracy, median task duration, tool-call count, and unrecoverable failure rate before any agent reaches production.
- Benchmark against your own SecOps stack Replay representative investigations from your SIEM, EDR, cloud, and identity telemetry so the eval reflects your connectors, schemas, and alert patterns.
- Constrain tool access by task scope Limit each agent to the minimum search, enrichment, and lookup actions needed for the workflow, and separate low-risk summarisation from higher-risk investigation paths.
Bottom line: SecOps benchmarking now needs to account for accuracy, cost, runtime stability, and tool discipline together, because headline scores do not capture operational readiness.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:29 pm
Model selection is becoming an access-control decision. Once an AI system can query telemetry, chain tools, and shape incident workflows, its model class affects operational risk in the same way that privilege scope affects a human analyst. Accuracy alone is insufficient because latency and reliability determine whether the agent can be trusted inside live security operations. Practitioners should treat model choice as part of the control plane, not as a procurement preference.
A few things that frame the scale:
- Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.
- Gartner predicts that by the end of 2026, 40% of enterprise apps will feature task-specific AI agents.
A question worth separating out:
Q: How should organisations decide between fast and accurate SecOps models?
A: Choose based on the workflow’s risk and urgency. Faster models can suit live triage and alert enrichment, while higher-accuracy models fit deeper investigations and harder analytic tasks. The decision should be driven by the control objective, not by a generic preference for the biggest model.
👉 Read our full editorial: SecOps agent evaluations show GPT-5 leading accuracy and cost