SecOps agent evaluations show GPT-5 leading accuracy and cost

At a glance

What this is: Cotool benchmarked frontier AI models on realistic SecOps tasks and found GPT-5 led in accuracy while Claude Haiku-4.5 led in speed.

Why it matters: Security teams evaluating AI for investigations, enrichment, and alert handling need to understand how accuracy, latency, cost, and tool reliability trade off before operationalising agents.

By the numbers:

• GPT-5 achieved the highest overall accuracy at 62.7%
• Claude Haiku-4.5 completed tasks the fastest at about 240 seconds on average
• Botsv3 comprises over 2.7M logs across 13 months
• All OpenAI and Anthropic models completed every question successfully at 100%

👉 Read Cotool's benchmark analysis of frontier models on SecOps tasks

Context

Agent evaluations matter because SecOps systems are not judged on intelligence in the abstract. They are judged on whether they can investigate, correlate, and answer correctly under tool constraints, noisy telemetry, and long multi-step workflows. In practice, model choice changes analyst throughput, detection quality, and the trust boundary around automated investigation.

This article is about how Cotool benchmarked frontier models on Splunk BOTSv3 using a purpose-built agent harness. The primary question is not whether AI can assist SecOps, but which model characteristics best fit different operational tasks such as triage, enrichment, summarisation, and deeper investigation. That is a typical problem for security teams moving from experimentation to controlled deployment.

The identity angle is indirect but real: once agents are connected to SIEMs, cloud telemetry, and identity providers, their access becomes an operational identity problem as much as an AI problem. The governance challenge is to decide what the agent may query, which tools it may invoke, and how much confidence to place in its outputs.

Key questions

Q: How should security teams choose AI models for SecOps workflows?

A: Choose models by task, not by brand. Use higher-accuracy models for deep investigations, faster models for interactive triage, and lower-cost models for summarisation or routine enrichment. Then validate the choice with repeatable evals that measure accuracy, latency, tool use, and failure recovery under realistic telemetry conditions.

Q: Why do SecOps agents need evaluation beyond accuracy scores?

A: Accuracy alone misses how an agent behaves in multi-step investigations. Security operations also depend on query efficiency, latency, reliability, and how often the agent fails mid-task. A model that answers correctly but stalls, over-calls tools, or breaks under connector errors can still be operationally unsuitable.

Q: What breaks when AI agent security is handled like ordinary application security?

A: Application security assumes a relatively stable workload boundary and a predictable request path. AI agents can select tools, access data, and continue executing in ways that change the path mid-workflow. When teams treat them like static apps, they miss the identity and authorisation layer where real risk appears.

Q: How do teams know if a SecOps agent is actually reliable?

A: Reliability shows up as stable task completion, consistent tool behaviour, and repeatable results across model versions and connector changes. Teams should look for low variance in runtime, controlled query counts, and high completion rates on representative tasks. If those metrics drift, the agent is not yet production ready.

Technical breakdown

How SecOps agent evals turn investigation quality into a measurable control

A SecOps eval converts an open-ended investigation task into a repeatable test by fixing the dataset, tool set, and expected answer. That lets teams score accuracy, latency, tool usage, and failure modes instead of relying on subjective demos. The hard part is realism: investigations are multi-step, non-deterministic, and often depend on external systems that can change between runs. A useful benchmark therefore needs a sandbox that approximates production telemetry while still producing stable measurements across model versions.

Practical implication: define benchmark tasks that mirror your real incident workflows, not generic chat prompts.

Why tool access changes agent performance in Splunk-style investigations

When an AI agent works against a SIEM, the model is not just generating text. It is deciding which queries to run, which datasets to inspect, and how to refine hypotheses based on intermediate results. Small differences in tool-calling behaviour can materially affect accuracy and cost because each extra query changes latency and can introduce more noise. In operational terms, tool orchestration becomes part of the model’s effective capability, which is why evals must measure tool efficiency alongside answer correctness.

Practical implication: measure tool calls per task and set guardrails for query volume before production use.

What performance-cost frontiers mean for security automation design

The benchmark shows that the best model for SecOps is not necessarily the fastest or the cheapest. Teams need to optimise for a mix of answer quality, runtime, and operating cost. A performance-cost frontier is the set of models that deliver the best tradeoff at different price points. That matters because some tasks, such as interactive triage, tolerate slightly lower depth if response time is short, while deeper investigations need higher accuracy even if they cost more.

Practical implication: map model choice to task class, then assign different models to triage, enrichment, and deeper investigation.

NHI Mgmt Group analysis

Model benchmarking has become a governance control, not just a lab exercise. Once security teams let AI agents touch telemetry, the benchmark becomes the evidence base for safe operational use. Accuracy alone is insufficient because tool reliability, latency, and failure recovery shape real analyst outcomes. For identity-linked workflows, the same logic applies to agent access into SIEMs, cloud consoles, and identity platforms. Practitioners should treat eval design as part of control validation, not post-hoc reporting.

SecOps agents create a new governance layer between human analysts and operational data. The agent is neither a passive model nor a fully autonomous operator, but it still needs scoped access, auditable behaviour, and clear limits on query action. That makes the access pattern itself an identity problem when agents query logs, cloud events, or identity provider data. The practical conclusion is that model selection and access design must be reviewed together, not as separate decisions.

Performance-cost tradeoffs will increasingly drive SOC automation architecture. The article shows that the best-performing model and the fastest model are not the same thing, which means teams need workload-specific routing. Alert enrichment, summarisation, and deep investigation should not share identical model assumptions or risk thresholds. The stronger the operational dependency, the more important it becomes to segment tasks by risk, latency tolerance, and verification depth.

Continuous evaluation is the only sustainable way to manage agent drift. As models, connectors, and detection rules change, an agent that worked well last month may degrade silently. This is especially important when the agent is connected to identity and cloud systems, where small errors can alter the trust boundary around investigations. Practitioners should build re-test cycles into release management so that agent quality is measured as continuously as the telemetry it consumes.

SecOps evaluation frameworks should align with security controls, not just machine learning metrics. NIST AI RMF is relevant here because it asks teams to govern, map, measure, and manage AI risk across the lifecycle. Where agents have access to production tools, the evaluation itself should prove that those access paths are bounded, monitored, and justified. The field should move toward control evidence that combines model performance with operational accountability.

What this signals

SecOps agent adoption will force security teams to formalise AI operational controls before broad rollout. Once an agent can query a SIEM or correlate cloud events, it inherits a privileged operational role that needs scope, logging, and measurable boundaries. The lesson for programmes is straightforward: do not move from proof-of-concept to production without an eval baseline, a rollback plan, and a defined trust model for the agent’s tool access.

Agent performance metrics will increasingly become procurement and governance evidence. Accuracy, latency, and task completion are now part of the control story because they determine whether an agent improves response quality or creates analyst risk. Teams should expect procurement, architecture, and GRC stakeholders to ask for benchmark evidence before approving AI-assisted investigations, especially where the agent touches identity or cloud telemetry.

For practitioners

• Define task-specific benchmarks Build evals from your own investigation patterns, such as phishing triage, cloud anomaly review, or identity-log correlation, so the benchmark reflects actual SecOps work. Use fixed datasets and answer keys to compare models consistently across releases.
• Measure tool efficiency alongside accuracy Track query count, average runtime, and task completion rate in addition to answer correctness. A model that is slightly less accurate but dramatically more efficient may be better for alert enrichment, while deeper investigations may justify slower execution.
• Scope agent access like any other privileged workload Give the agent only the SIEM indexes, identity sources, and cloud telemetry it genuinely needs, then log every query and tool invocation. Treat the agent as a privileged workload with accountable access rather than an open-ended analyst surrogate.
• Re-run evals on every meaningful change Repeat benchmark suites when the model version changes, a connector schema shifts, or a detection rule is updated. This catches drift before it appears in production and helps you separate model performance from system design effects.

Key takeaways

• SecOps AI is no longer a general model-selection problem. It is a control problem defined by task fit, tool access, and measurable reliability.
• The benchmark shows that the best answer, the fastest answer, and the cheapest answer are not the same thing, so programme design must separate triage from deep investigation.
• Teams that want to operationalise agents need eval baselines, scoped access, and continuous re-testing or they will inherit silent drift.

Key terms

• SecOps Agent: A SecOps agent is an AI system that uses security tools to investigate alerts, query telemetry, and assist with operational tasks. Its value depends on repeatable performance, bounded access, and auditable behaviour, not just model capability.
• Evaluation Harness: An evaluation harness is the test environment, scripts, permissions, and network setup used to assess a model or system before release. Its security matters because the harness is often the difference between harmless testing and a real-world compromise path if it is allowed to reach external systems.
• Performance-Cost Frontier: A performance-cost frontier describes the set of model choices that offer the best tradeoff between quality and spend. For security teams, it helps separate the model that is cheapest from the one that is actually efficient for a given task.

What's in the full report

Cotool's full research covers the operational detail this post intentionally leaves for the source:

• Benchmark methodology for the Splunk BOTSv3 environment, including the full dataset, tool set, and answer validation approach
• Per-model results tables showing accuracy, completion rate, latency, tool calls, and token usage across frontier model families
• Follow-up analysis plans for Gemini failure modes and long-running agent trajectories that were not fully unpacked here
• The expansion path into cloud telemetry, identity providers, and endpoint platforms for broader SecOps evaluation

👉 Cotool's full research includes the methodology, model-by-model performance data, and planned follow-up analysis.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, identity lifecycle, secrets management, and agentic AI identity. It helps security and identity practitioners build the control foundations needed when automated systems connect to sensitive tools and data.