SecOps agent benchmarks: what do the BOTSv3 results mean?

TL;DR: A benchmark of frontier models on realistic SecOps tasks found GPT-5 reached 62.7% accuracy, Claude Haiku-4.5 was the fastest at about 240 seconds, and GPT-5 variants occupied the strongest performance-cost frontier, according to Cotool. The results suggest enterprise teams need to treat model selection as an operational control, not a generic AI choice.

NHIMG editorial — based on content published by Cotool: LLMjacking and SecOps benchmark analysis using BOTSv3

By the numbers:

• Botsv3 comprises over 2.7M logs across 13 months

Questions worth separating out

Q: How should security teams choose AI models for SecOps workflows?

A: Choose models by task, not by brand.

Q: Why do SecOps agents need evaluation beyond accuracy scores?

A: Accuracy alone misses how an agent behaves in multi-step investigations.

Q: What breaks when AI agent security is handled like ordinary application security?

A: Application security assumes a relatively stable workload boundary and a predictable request path.

Practitioner guidance

• Define task-specific benchmarks Build evals from your own investigation patterns, such as phishing triage, cloud anomaly review, or identity-log correlation, so the benchmark reflects actual SecOps work.
• Measure tool efficiency alongside accuracy Track query count, average runtime, and task completion rate in addition to answer correctness.
• Scope agent access like any other privileged workload Give the agent only the SIEM indexes, identity sources, and cloud telemetry it genuinely needs, then log every query and tool invocation.

What's in the full report

Cotool's full research covers the operational detail this post intentionally leaves for the source:

• Benchmark methodology for the Splunk BOTSv3 environment, including the full dataset, tool set, and answer validation approach
• Per-model results tables showing accuracy, completion rate, latency, tool calls, and token usage across frontier model families
• Follow-up analysis plans for Gemini failure modes and long-running agent trajectories that were not fully unpacked here
• The expansion path into cloud telemetry, identity providers, and endpoint platforms for broader SecOps evaluation

👉 Read Cotool's benchmark analysis of frontier models on SecOps tasks →

SecOps agent benchmarks: what do the BOTSv3 results mean?

Explore further

View Full Forum →  |  NHI Foundation Course →