AI threat detection still depends on human review and context

At a glance

What this is: This is an analysis of how AI threat detection works in the SOC and where it still fails without human review, especially for context-heavy triage and containment decisions.

Why it matters: It matters to IAM practitioners because identity, credential abuse, and privileged response actions are now central to AI-assisted detection workflows, and those controls fail if automation is allowed to act without governance.

By the numbers:

• Organizations now process an average of 960 security alerts daily, and nearly half go uninvestigated.
• 67% of organizations are now shifting to behavior-based detection.
• 90 zero-day vulnerabilities were exploited in the wild in 2025, with 48% targeting enterprise technologies.
• The total detection-and-containment timeline drops from 299 days without extensive AI deployment to 201 days with extensive AI deployment, a 98-day difference.

👉 Read Panther's analysis of AI threat detection, human review, and SOC workflow design

Context

AI threat detection is the use of machine learning to identify suspicious activity by comparing live behavior with learned baselines rather than relying only on static signatures. That shift matters because modern security environments generate more alerts than human teams can reasonably review, and the article frames the real issue as workflow design, not model novelty.

The article also points to a clear identity-security intersection: compromised accounts, lateral movement, and privileged response actions are now part of the same detection chain. For IAM, PAM, and NHI programmes, the question is whether AI tools are being used to support accountable identity governance or to automate decisions that still need human context.

Key questions

Q: How should security teams use AI without turning it into a control dependency?

A: Security teams should use AI for summarisation, correlation, and prioritisation, then keep containment in deterministic controls such as access policy, segmentation, and revocation. The key rule is that AI can recommend action, but it should not be the only mechanism that can stop exposure. That separation reduces false confidence and preserves auditability.

Q: Why do identity and privilege signals matter so much in AI threat detection?

A: Because many real incidents move through valid credentials, delegated access, and over-privileged accounts rather than obvious malware. If a detection platform cannot interpret identity context, it will miss the difference between expected access and abuse. That gap is especially important in cloud and NHI-heavy environments.

Q: What do teams get wrong about anomaly detection in operations?

A: They often assume an anomaly score is the same as a validated incident. It is not. Anomaly detection is a prioritisation tool that helps narrow attention, but it still depends on good baselines, clean telemetry, and operator judgment. Without those, teams can automate around noise instead of around real operational risk.

Q: How can organisations tell whether their AI security model is actually working?

A: They should test whether the control stack can explain who acted, what data was touched, and what purpose the action served. If those three signals cannot be correlated in one incident view, the model is likely monitoring access without governing behaviour. That is a visibility gap, not a complete AI security posture.

Technical breakdown

Behavioural baselines and anomaly scoring in AI threat detection

AI threat detection works by learning what normal activity looks like for users, devices, applications, and network flows, then scoring deviations in real time. Supervised models classify known patterns, while unsupervised models surface novel outliers. The output is usually a ranked signal, not proof of malicious activity. This is why the quality of telemetry, schema normalisation, and baseline windows matters more than the model label itself. If the data feeding the system is incomplete or noisy, the score can be precise-looking but operationally weak.

Practical implication: validate telemetry coverage and baseline quality before trusting anomaly scores for identity or access decisions.

AI agents in SOC triage: context gathering versus final decisions

AI agents are useful in triage because they can collect context from logs, enrich alerts, and produce structured investigation summaries quickly. They are less suitable for decisions that depend on organisational context, such as change windows, business criticality, or incident blast radius. That separation is important: the agent can accelerate analysis, but it cannot own accountability for containment. In practice, the strongest deployments use AI to reduce analyst workload while preserving human approval for actions that can affect production systems or critical accounts.

Practical implication: define escalation gates so AI can assist with triage but cannot close or contain high-impact identity incidents alone.

Why AI detection still needs human review for adversarial inputs

Detection systems themselves are part of the attack surface. Adversaries can manipulate models through adversarial input, poisoning, or noise that causes blind spots and false negatives. This is especially relevant in security operations because a silent model can look healthy while missing the very technique an attacker is using. Human review remains necessary to compare threat intelligence with detection coverage and to write compensating rules when the model does not alert. MITRE ATLAS is the clearest reference point for these AI-specific attack techniques.

Practical implication: treat model bypass and poisoning as active threats and test detection logic against adversarial cases.

Threat narrative

Attacker objective: The attacker’s objective is to operate inside the environment long enough to evade detection, preserve access, and complete data theft or lateral movement without triggering timely response.

1. Entry occurs when an attacker uses compromised credentials or identity abuse to generate activity that looks normal enough to evade simple signature matching.
2. Escalation follows as the attacker moves laterally, reuses access, or manipulates model inputs so the detection system misclassifies the behaviour or stays silent.
3. Impact is reached when the alerting pipeline misses the intrusion or delays containment long enough for account compromise, exfiltration, or service disruption to spread.

NHI Mgmt Group analysis

Behaviour-based detection is now an identity problem as much as a SOC problem: once stolen credentials become the second most common initial access vector, identity telemetry becomes detection telemetry. That means IAM, PAM, and NHI governance can no longer sit beside SOC analytics as separate programmes. They have to share baselines, alert logic, and escalation criteria. The practical conclusion is that identity context must be built into threat detection design.

Human review remains the control boundary, not a legacy inconvenience: the article is right to separate triage acceleration from containment authority. AI can compress analyst workload, but it cannot own the business impact judgment that determines whether a lockout, isolation, or privilege revoke is acceptable. That makes approval gates a governance control, not an efficiency tax. Practitioners should treat human sign-off as part of the detection architecture.

Detection logic stored as code is the right direction, but only if it is testable and reviewable: readable rules, reproducible workflows, and versioned logic reduce operational drift. This matters because model-led security quickly becomes un-auditable when teams cannot explain why a signal fired or why an action was taken. The broader lesson is that AI in security must remain inspectable to stay governable.

Adversarial manipulation is the named concept security teams should adopt here: AI detection does not just miss attacks, it can be shaped by them. Prompt abuse, poisoning, and chaff data are not theoretical edge cases in a SOC context. They are governance failures when teams assume the model is an observer rather than a target. Practitioners should build detection controls with model abuse explicitly in scope.

What this signals

The next operational shift is toward identity-aware detection pipelines, where identity context, privilege scope, and service account behaviour are first-class inputs to the SOC. That matters because the same compromised account can appear benign to a generic model and suspicious to an IAM-aware one. Teams should align detection engineering with NIST Cybersecurity Framework 2.0 and keep the identity layer visible in incident triage.

AI-assisted triage will keep improving, but the governance burden will move upward rather than disappear. As AI agents take on more enrichment work, security leaders will need clearer audit trails, model test cases, and approval boundaries for any action that can interrupt service or alter access. The practical signal is simple: if your SOC cannot explain why an AI recommendation was made, it cannot safely automate around it.

The strongest programmes will connect behaviour analytics to identity standards and adversarial AI threat modelling, using resources such as the NIST Cybersecurity Framework 2.0 and the MITRE ATLAS adversarial AI threat matrix. That combination helps teams see both sides of the problem, the compromised identity and the model that may miss it. The result is better coverage for the attacks most likely to move faster than human review.

For practitioners

• Build identity-aware detection baselines Feed the SOC with identity, PAM, and NHI signals alongside endpoint and cloud telemetry so anomaly scoring can distinguish routine admin activity from compromised access patterns. Prioritise accounts with privileged reach and service accounts with broad lateral movement potential.
• Set human approval gates for disruptive actions Require analyst approval before account disables, host isolation, network segmentation changes, or any response that can interrupt production services. Use explicit thresholds for when AI can recommend a control and when a person must authorise it.
• Version and test detection logic as code Store detection rules in a reviewable codebase, track changes, and test them against known adversarial cases so alert logic can be audited and tuned without guesswork. This is especially important when baselines shift after cloud or identity changes.
• Exercise model-bypass scenarios in tabletop tests Include poisoning, chaff data, and living-off-the-land behaviour in purple-team scenarios so the team can see where AI detection stays silent. Compare those outcomes against MITRE ATLAS techniques and update compensating rules accordingly.

Key takeaways

• AI threat detection improves SOC speed, but it does not remove the need for human judgment on high-impact identity actions.
• Identity, credential abuse, and model manipulation now sit inside the same detection problem, which makes governance as important as model accuracy.
• The most reliable programmes will combine readable detection logic, approval gates, and adversarial testing rather than treating AI as an autonomous responder.

Key terms

• Behavior Baseline: A record of normal activity for a non-human identity, including typical consumers, resources, and actions over time. Baselines help security teams detect when an identity is being used in an unusual way and provide the context needed to enforce least privilege safely in dynamic environments.
• Human Approval Gate: A human approval gate is a control point where a person must confirm an action before it proceeds. For AI agents, it is the clearest boundary between assistance and execution, especially when the action can move money, expose data, or trigger irreversible change.
• Adversarial Input: Adversarial input is data crafted to mislead a model into the wrong conclusion or no conclusion at all. In security operations, that can mean chaff, poisoning, or carefully shaped activity designed to hide malicious behaviour from a detection system.
• Detection Logic as Code: Detection logic as code means writing, versioning, reviewing, and testing detection rules in a source-controlled format. It makes alerting explainable and auditable, which is essential when teams need to prove why a model fired and how a response decision was made.

What's in the full article

Panther’s full blog covers the operational detail this post intentionally leaves for the source:

• A deeper walkthrough of how Panther structures AI SOC workflows across ingest, triage, and escalation.
• Examples of detection logic and rule design that show how the platform packages signals for analysts.
• Operational examples of where human review is still required before account disables, isolation, or other disruptive actions.
• Platform-specific details on detection logic stored in Python or YAML for teams already standardising on code-first workflows.

👉 Panther's full post covers the detection pipeline, AI agent triage, and the human approval gates needed for production use.

Deepen your knowledge

NHI Mgmt Group’s NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle controls. It is designed for practitioners who need to connect identity governance to broader security operations and risk decisions.