SecOps agent evaluations show GPT-5 leading accuracy and cost

At a glance

What this is: Cotool benchmarked frontier AI models on realistic SecOps tasks and found GPT-5 delivered the highest accuracy while GPT-5 variants dominated the performance-cost frontier.

Why it matters: Security teams evaluating AI agents for investigations, alert enrichment, or log analysis need to treat model selection, latency, and tool reliability as governance decisions, not just benchmark scores.

By the numbers:

• GPT-5 achieved the highest overall accuracy at 62.7% on the BOTSv3 benchmark.
• Claude Haiku-4.5 completed tasks the fastest at about 240 seconds on average.
• BOTSv3 comprises over 2.7M logs spanning over 13 months and 59 question and answer pairs.

Context

SecOps agent evaluation is becoming a governance problem, not just a model-testing exercise. When an AI system is asked to investigate security events, success depends on more than raw answer quality. Latency, tool discipline, and completion reliability all shape whether the agent can be trusted in operational workflows. For security teams, the key question is how to measure those behaviours before an agent touches real telemetry or incident queues.

This benchmark uses the BOTSv3 blue-team challenge to compare frontier models on realistic security operations tasks. That matters because SecOps work blends search, context gathering, and decision-making across noisy logs and changing environments. The identity angle is indirect but real: as agents begin interacting with cloud telemetry, identity provider data, and endpoint platforms, they start behaving like governed non-human operators inside the security stack.

The article’s core message is typical of the current market: benchmark scores alone are not enough to justify deployment. Teams need evidence about accuracy, cost, speed, and failure modes under conditions that resemble their own detection and investigation workflows.

Key questions

Q: How should security teams evaluate SecOps agents before putting them near production data?

A: Use task-level evals that mirror real investigations, then measure accuracy, completion rate, latency, and tool behaviour together. A model that answers correctly but times out, over-calls tools, or behaves inconsistently is not ready for production support. The safest approach is to benchmark against your own SIEM, EDR, cloud, and identity workflows.

Q: Why does a high benchmark score not guarantee safe AI use in SecOps?

A: Because SecOps value depends on more than getting the final answer right. Teams also need predictable tool use, bounded access, and acceptable runtime behaviour. A model that is accurate in a benchmark can still be operationally risky if it fails unpredictably, consumes too many calls, or cannot sustain long investigations.

Q: What do security teams get wrong about AI-assisted investigations?

A: They assume the model is the main value. In practice, the value comes from the quality and accessibility of the underlying data plus the consistency of the investigation method. If those are weak, AI simply automates confusion. The right goal is to scale expert judgment, not to replace evidence quality with faster output.

Q: How should organisations decide between fast and accurate SecOps models?

A: Choose based on the workflow’s risk and urgency. Faster models can suit live triage and alert enrichment, while higher-accuracy models fit deeper investigations and harder analytic tasks. The decision should be driven by the control objective, not by a generic preference for the biggest model.

Technical breakdown

Why SecOps agents need task-specific evaluation

An evaluation in agentic security is a controlled test of whether a system can complete a specific operational task under defined constraints. In this case, the task is not general reasoning but multi-step SecOps work over Splunk data and question-answer pairs. That distinction matters because an agent can look strong in broad benchmarks yet struggle when it must search logs, preserve context, and use tools correctly. Realistic evaluation also exposes trade-offs between accuracy, tool use, and latency that synthetic tests miss.

Practical implication: benchmark agents against your own security workflows and judge them on task completion, not model reputation.

What BOTSv3 reveals about log-driven investigation workflows

BOTSv3 is a blue-team capture-the-flag environment with millions of logs and security questions that require iterative investigation. A setup like this reflects how SecOps actually works: the agent has to navigate a dataset, form hypotheses, run searches, and extract the right answer from noisy evidence. This makes it a better proxy for operational use than single-prompt classification. It also shows why connector quality matters, because the agent’s performance depends on how cleanly it can access SIEM data and related sources.

Practical implication: test AI agents in environments that mirror your SIEM and investigation flow, not in isolated prompt exercises.

Performance-cost frontier in agentic security

The performance-cost frontier is the set of models that give the best trade-off between accuracy and cost. In operational security work, that trade-off is not abstract. A slightly slower or more expensive model may still be the right choice for high-stakes investigations, while cheaper models may fit summarisation or continuous triage. The benchmark also shows why completion reliability and call behaviour matter, because a model that fails less often or uses tools more consistently can be safer to operationalise than one that merely scores well on paper.

Practical implication: select models by the control they will support, and match cost, speed, and reliability to the workflow’s risk level.

Threat narrative

Attacker objective: The objective is not a breach but operational control, meaning the ability to complete security investigations accurately enough to support real SecOps decisions.

1. Entry occurs when an agent is given access to security telemetry and investigation tools inside a live or sandboxed SecOps environment.
2. Credential and privilege exposure arise when the agent is allowed to query log sources and related systems with enough access to complete multi-step investigations.
3. Impact is measured as whether the agent can produce correct answers quickly, reliably, and at acceptable cost without drifting or failing under tool pressure.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Model selection is becoming an access-control decision. Once an AI system can query telemetry, chain tools, and shape incident workflows, its model class affects operational risk in the same way that privilege scope affects a human analyst. Accuracy alone is insufficient because latency and reliability determine whether the agent can be trusted inside live security operations. Practitioners should treat model choice as part of the control plane, not as a procurement preference.

Security teams are now evaluating non-human operators, not just software features. A SecOps agent behaves like a governed workload with changing behaviour, external tool access, and measurable task outcomes. That makes it closer to a non-human identity governance problem than a generic chatbot deployment. The right question is not whether a model can answer questions, but whether its access, tool use, and error profile are acceptable for the workflow it will inhabit.

Tool discipline matters as much as benchmark accuracy. The article shows that a model can be accurate, fast, or cheap, but the operational winner is the one whose behaviour stays predictable across long runs and multiple tool calls. That is the emerging governance lesson for agentic AI security: stable execution, bounded access, and measurable completion rates define readiness. Practitioners should insist on runtime evidence before allowing AI to influence security decisions.

SecOps benchmarking is shifting toward continuous assurance. A one-time eval is not enough when models, connectors, and workflows change constantly. The article points toward a future where agent quality is re-tested whenever the model version, connector schema, or detection rule changes. That is a stronger assurance posture for agentic AI than static approval, and it should be the baseline for any team considering security automation at scale.

From our research library:

• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.
• Gartner predicts that by the end of 2026, 40% of enterprise apps will feature task-specific AI agents.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Continuous evaluation is now a governance requirement for SecOps agents. As models, connectors, and detection content change, yesterday’s benchmark result becomes a weak proxy for tomorrow’s behaviour. Security teams need repeatable testing that treats AI agents as operational systems with measurable drift, not static products.

Identity context will matter as these agents expand beyond SIEM. Once the same agent starts touching cloud telemetry, identity providers, and endpoint platforms, access scope becomes part of the risk model. That makes non-human identity governance relevant to agentic security even when the original use case begins in SecOps.

The practical shift is toward runtime assurance rather than one-off approval. Teams should expect to validate tool access, completion reliability, and workflow-specific outcomes whenever a model, connector, or detection rule changes.

For practitioners

• Define workflow-specific acceptance thresholds Set separate pass criteria for investigation accuracy, median task duration, tool-call count, and unrecoverable failure rate before any agent reaches production.
• Benchmark against your own SecOps stack Replay representative investigations from your SIEM, EDR, cloud, and identity telemetry so the eval reflects your connectors, schemas, and alert patterns.
• Constrain tool access by task scope Limit each agent to the minimum search, enrichment, and lookup actions needed for the workflow, and separate low-risk summarisation from higher-risk investigation paths.
• Track completion reliability over raw score Monitor how often the agent finishes without timeout, tool failure, or drift, because a high accuracy score is less useful if completion is inconsistent.
• Re-run evals on every meaningful change Treat model upgrades, connector changes, and detection-rule updates as triggers for repeat testing so you do not inherit stale performance assumptions.

Key takeaways

• SecOps benchmarking now needs to account for accuracy, cost, runtime stability, and tool discipline together, because headline scores do not capture operational readiness.
• The article shows that model choice changes the control profile of an AI-enabled security workflow, especially when the agent is allowed to search, enrich, and investigate across live telemetry.
• Continuous re-testing is becoming necessary as models and connectors evolve, which means AI governance for SecOps will need ongoing assurance rather than a single launch decision.

Key terms

• SecOps Agent: A SecOps agent is an AI system that uses security tools to investigate alerts, query telemetry, and assist with operational tasks. Its value depends on repeatable performance, bounded access, and auditable behaviour, not just model capability.
• Performance-Cost Frontier: A performance-cost frontier describes the set of model choices that offer the best tradeoff between quality and spend. For security teams, it helps separate the model that is cheapest from the one that is actually efficient for a given task.
• Integration Reliability: Integration reliability is the ability of a connector or sync layer to keep reading and writing identity data accurately under production conditions. In identity security, it is not enough for an integration to exist. It must continue to handle rate limits, schema changes, retries, and failures without producing silent gaps.
• Evaluation Harness: An evaluation harness is the test environment, scripts, permissions, and network setup used to assess a model or system before release. Its security matters because the harness is often the difference between harmless testing and a real-world compromise path if it is allowed to reach external systems.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps practitioners design controls for non-human systems that now operate inside security workflows.