Notifications
Clear all
Topic starter
20/08/2026 6:10 pm
TL;DR: AI agents are being used for threat hunting, with automatic and co-pilot modes that depend on strong evaluation, trace visibility, and human annotation to keep black-box behaviour from reaching security operations, according to Arize. The governance challenge is no longer whether agents can search faster, but whether their reasoning, tool use, and outputs remain auditable enough for operational trust.
NHIMG editorial — based on content published by Arize: How Nebulock Democratizes Threat Hunting
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: How should security teams govern agentic AI as it moves into production?
A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature.
Q: Why do agentic AI security workflows need identity and access controls?
A: Because the model itself is not the whole system.
Q: What breaks when AI agent access is not re-evaluated in real time?
A: The main failure is privilege drift.
Practitioner guidance
- Instrument agent traces end to end Capture tool calls, prompt changes, retrieved evidence, and reasoning steps so investigators can replay why the agent reached a conclusion.
- Separate read-only hunting from action-capable workflows Keep co-pilot investigation privileges distinct from any workflow that can modify detections, tickets, or environment state.
- Gate releases with evals and golden datasets Test prompt edits, tool additions, and orchestration changes against curated threat-hunting cases before deployment.
What's in the full article
Arize's full interview covers the operational detail this post intentionally leaves for the source:
- How Nebulock uses human annotations and LLM-as-a-judge methods to evaluate both outputs and reasoning steps.
- Why experiment tracking and golden datasets matter when prompts, tools, and agent behaviour change over time.
- How production traces are organised and monitored across the development and deployment lifecycle.
- Where the team sees multi-agent iteration creating hidden regressions in customer-facing security workflows.
👉 Read Arize's interview on how Nebulock uses AI agents for threat hunting →
AI threat hunting agents: can your evals keep up with production?
Explore further
20/08/2026 6:46 pm
Agentic threat hunting turns AI observability into a governance requirement. The important question is no longer whether an AI system can surface threats, but whether its reasoning can be inspected, replayed, and challenged. That is a control problem, not just a model-quality problem. In AI operations, traceability is the difference between useful automation and ungoverned decision support. Practitioners should treat internal reasoning visibility as part of the security control set.
A question worth separating out:
Q: How do teams know an AI hunting co-pilot is actually working?
A: Look for decision quality, not activity volume. Good signals include repeatable trace quality, fewer dead-end investigations, lower analyst rework, and consistent results on golden datasets. If the agent cannot explain its path through evidence or its outputs vary wildly across similar cases, it is not ready for operational trust.
👉 Read our full editorial: Agentic threat hunting raises the bar for evaluation and trust
