AI is changing security operations, but human control still matters

At a glance

What this is: This is an analysis of where AI is already improving security operations, especially triage, hunting, prioritisation, incident response, and phishing defence, while exposing new governance risks around hallucination and over-automation.

Why it matters: It matters because security teams are starting to operationalise AI inside SOC and IR workflows, which changes how practitioners validate outputs, manage sensitive data, and decide which actions can be automated safely.

Context

AI in security operations is becoming a control and governance issue, not just a tooling question. Large language models, embeddings, and machine learning pipelines can now help analysts reduce noise, but they can also introduce false confidence, data handling risk, and automation mistakes if humans are removed from the decision path.

The identity angle is real because these workflows increasingly touch endpoints, cloud telemetry, identity data, and privileged actions. When AI helps triage or respond, practitioners still need control over what the system sees, what it recommends, and what it is allowed to execute; otherwise AI becomes another source of operational drift rather than resilience.

Key questions

Q: How should security teams use AI in the SOC without losing human control?

A: Use AI to remove repetitive work, enrich alerts, and accelerate triage, but keep humans accountable for escalation, containment, and exception handling. The right model is human-centred automation, where AI expands analyst capacity without becoming the final decision-maker for high-risk actions. That requires explicit approval gates, audit trails, and ownership for every automated step.

Q: Why do AI models create risk in incident response and remediation?

A: Because they can sound confident while inventing details such as log fields, CVE references, or fix steps. In an incident, that can push responders toward the wrong containment decision or an unnecessary service disruption. The answer is to ground outputs in retrieved evidence, require analyst verification, and limit automation to reversible, low-impact actions.

Q: What do security teams get wrong about using AI agents for threat hunting?

A: They often assume the agent is the source of insight. In practice, the insight comes from human context, and the agent only scales that context across more data. Without well-curated TTP knowledge, agents will produce noise, miss subtle variants, or overfit to weak signals.

Q: What happens when AI-driven security automation is introduced without human oversight?

A: AI-driven security automation without human oversight can create bad response decisions, missed context, and unnecessary disruption if the system acts on incomplete evidence. The safer model is human-in-the-loop: let AI accelerate enrichment, investigation, and workflow initiation, then keep accountability with the SOC for escalation and final action. That balance preserves speed while reducing operational risk.

Technical breakdown

AI-assisted alert triage and SOC summarisation

Modern SOC tooling uses classifiers, embeddings, and language models to reduce alert noise by clustering similar events, suppressing duplicates, and adding context at triage time. This is not autonomous detection. It is decision support that shortens the path from raw telemetry to an analyst-readable summary. The real architectural change is that the model sits between data ingestion and human review, turning many low-value records into a smaller set of prioritised work items. That can materially improve throughput, but only if the underlying telemetry is trustworthy and the model output is treated as a draft, not an authority.

Practical implication: keep a human approval step for escalations that affect containment, blocking, or incident declaration.

Attack path prioritisation is more useful than flat vulnerability lists

AI adds value when it correlates scan results, asset inventories, identity graphs, and exploit intelligence into attack paths instead of long vulnerability backlogs. The security problem is not the existence of thousands of findings. It is knowing which combination creates a credible route to a sensitive asset. That requires contextual reasoning about reachability, privilege, and chaining, which is where machine learning can surface the few paths worth fixing first. This is especially relevant in environments where identities, cloud resources, and code repos are interdependent and where the most dangerous issue is often the chain, not the single finding.

Practical implication: prioritise remediation by exploit path and exposed privilege, not by CVSS alone.

Why hallucination makes AI a governance problem

Language models can invent CVE numbers, log fields, or remediation steps with confidence, which makes unchecked output dangerous in operational security. The mechanism matters because incident response and blocking decisions rely on accuracy under pressure. If the model is not grounded in retrieved evidence, it can accelerate the wrong action faster than a human could have made the same mistake manually. In practice, that means AI output needs provenance, source retrieval, and verification before it drives containment or remediation. The governance issue is not whether the model is useful. It is whether the organisation can prove when it is safe to trust it.

Practical implication: require evidence-backed outputs for any AI recommendation that changes production state.

Threat narrative

Attacker objective: The attacker aims to gain a faster, more reliable path into target environments while defenders are distracted by high-volume, low-confidence AI-assisted workflows.

1. Entry begins with AI-assisted phishing and industrialised reconnaissance that improves lure quality and target selection.
2. Credential access and initial foothold become easier when attackers use generated content, scraped context, and automated workflow support to increase click-through and response rates.
3. Escalation and impact follow when defenders over-trust AI outputs, automate too aggressively, or fail to validate model-generated recommendations before acting.

Breaches seen in the wild

• reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.
• Shai Hulud npm malware campaign: Shai Hulud campaign: npm malware exposed secrets on GitHub.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI in security operations is a decision-support layer, not a decision owner. The article shows that AI can compress triage, hunting, and incident analysis, but the security value comes from accelerating human judgment rather than replacing it. That distinction matters because SOC and IR teams still carry accountability for containment, prioritisation, and escalation. Practitioners should therefore design AI as a governed augmentation layer, not as an autonomous operator.

Grounding is the new control plane for AI-assisted operations. The article’s biggest operational warning is hallucination, which turns model output into a trust question rather than a productivity question. The named concept here is grounded response drift: when an AI system produces plausible but unverified operational guidance that pulls response off course. That risk is especially acute in security workflows where a false log field or bad remediation step can change production state. Practitioners should treat evidence retrieval and verification as mandatory control points.

AI increases the speed of both defence and attack, which raises the value of validated exposure data. The article ties defensive prioritisation to realistic attack paths and continuous validation, which is the right model for modern operations. If the exposure picture is stale, AI will simply help teams be wrong more efficiently. Security leaders should use AI to rank what matters, but they must keep continuous testing and attack-path validation in place to keep those rankings honest.

Identity and privilege data are becoming core inputs to AI security operations. The article explicitly connects prioritisation to identity graphs and privileged pathways, which means AI-assisted defence now depends on how accurately organisations model access and trust. That places IAM, PAM, and machine identity governance directly inside the operational loop. Practitioners should expect AI value to rise only when identity data is current, scoped, and audit-ready.

Over-automation is the easiest way to turn a useful model into an incident amplifier. The article is clear that full automation should stay limited to low-risk, high-confidence actions. Once AI is allowed to quarantine systems, block users, or roll back changes without robust guardrails, the operational blast radius can exceed the original event. Teams should define explicit human-in-the-loop thresholds before they expand AI action rights.

What this signals

Grounding is now the governing constraint for AI-assisted security operations. The practical question is no longer whether AI can summarise or prioritise, but whether the organisation can prove that each recommendation is tied to evidence the operator can inspect. That is where AI security moves from experimentation to control design, and where identity, telemetry, and response workflows start to converge.

Identity data has become an operational input to security AI, not just an audit record. When attack-path ranking depends on privileges, access scope, and identity relationships, stale IAM and PAM data degrades the quality of every downstream decision. Teams that want AI to improve prioritisation need current identity state as part of the control plane, not as a reporting afterthought.

For practitioners

• Define human-in-the-loop thresholds for AI-assisted response Require manual approval for containment, isolation, access blocking, and production rollback whenever model confidence, blast radius, or business impact is material.
• Ground AI outputs in retrieved evidence Use retrieval-backed summaries, source citations, and analyst verification before any AI recommendation is used in triage or remediation.
• Prioritise attack paths over vulnerability counts Rank remediation by reachable attack chains that combine exposure, privilege, and sensitive assets instead of by raw CVSS volume.
• Keep identity and privilege data current Feed AI prioritisation engines with accurate identity graphs, privilege scopes, and machine access inventories so they reflect real blast radius.
• Separate low-risk automation from high-impact action Allow only bounded, reversible tasks to execute automatically and reserve high-impact security actions for human confirmation.

Key takeaways

• AI can materially improve security operations, but only when it shortens human decision-making rather than replacing it.
• The highest-value use cases are alert triage, attack-path prioritisation, incident summarisation, and phishing defence, not open-ended autonomy.
• Security leaders should treat grounding, verification, and human approval thresholds as core controls for AI-assisted workflows.

Key terms

• AI-Assisted Triage: The use of machine-driven prioritisation to sort, rank or route suspicious cases for human review. It can improve speed and consistency, but only if analysts can understand, challenge and override the recommendation. Without governance, it becomes a hidden decision layer inside the investigation process.
• Grounded Response: An AI output that can be traced to verifiable source material the system actually retrieved during the interaction. Grounding reduces the chance of invention, but only when retrieval, access controls, and refusal behaviour are designed to stop unsupported claims from being surfaced.
• Attack-path prioritization: Attack-path prioritization is the practice of ranking findings by whether they are actually reachable from an attacker’s likely path. It moves teams away from treating every vulnerability equally and toward fixing the issues that connect exposure, privilege, and sensitive data in a realistic compromise chain.
• Human-in-the-Loop (HITL): A governance pattern requiring human approval before an AI agent takes high-impact, irreversible, or out-of-scope actions. HITL is a critical control for agentic AI identity governance.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect access control and identity lifecycle decisions to broader security operations and governance.