What breaks is the ability to create useful detections for unfamiliar threats. AI can reuse known logic, but it struggles to decide which signals matter when the attack pattern is new or poorly documented. Without human analysts, the SOC risks encoding old assumptions into new workflows and missing the behaviours that actually matter.
Why AI Falls Short on New Detection Problems
Detection engineering is not only pattern generation. It is judgement about which events matter, which telemetry is trustworthy, and which behaviours deserve a rule even when there is no prior playbook. That is why the weak point is not speed, but the ability to reason through novelty, ambiguity, and incomplete evidence.
When the threat is familiar, AI can accelerate drafting and reuse existing logic. When the threat is novel, the work changes: analysts must decide whether a signal is meaningful, whether the environment has enough visibility, and whether a detection should express behaviour, not just known indicator matches. For a practical baseline on this discipline, see the Detection Engineering Guide.
This is also where alert quality becomes a design constraint. If the workflow only replays prior patterns, it can produce polished detections that miss the new attack path entirely. Good detection work therefore depends on understanding attack behaviour, not simply automating rule syntax.
Why Old Assumptions Become a Blind Spot
The failure mode is assumption lock-in. An AI-assisted process tends to favour what is already represented in its training data, prompt context, or prior detections, so it can overweight familiar telemetry and underweight unfamiliar but important signals. That matters most when an adversary changes tactics faster than the detection library can be updated.
Human analysts help by challenging the inherited model of what “normal” and “suspicious” look like. They notice when a workflow silently excludes weak signals, when a rule family is too dependent on a single log source, or when an apparent absence of alerts reflects a coverage gap rather than a clean environment. That judgment is central to using the SIEM Guide effectively, because SIEM value depends on what is onboarded, retained, tuned, and correlated.
There is a second blind spot: AI often optimises for plausible output, not for detection completeness. A rule that looks coherent can still be brittle if it anchors on the wrong field, the wrong precursor, or a too-specific event sequence. This is why detection engineering still needs explicit coverage thinking, not just content generation.
What Practitioners Need to Keep in the Loop
Detection engineering works best when AI is used as an accelerator for drafting, testing, and summarising, while analysts remain responsible for threat interpretation and coverage decisions. The most valuable human step is deciding whether the environment has evidence for the behaviour you care about, and if not, what telemetry or analytic change is needed before the rule can be trusted.
For teams building repeatable content, the right quality bar is not “did the model produce a rule?” but “did the rule survive validation against real telemetry and known edge cases?” A useful companion for that workflow is the Detection as Code Guide, because versioning, testing, and rollback make the difference between fast output and dependable detections.
When the attack pattern is unfamiliar, the practitioner judgement that matters most is whether the team can expand detection from indicator matching to behaviour modelling without losing precision. That usually means rechecking log sources, validating assumptions, and measuring whether the rule finds the right class of activity instead of just the easiest-to-recognise sample.
Risk and Threat Considerations
Leaving detection engineering entirely to AI creates two linked risks: the SOC may miss genuinely new behaviours, and it may overfit to old patterns that no longer represent the current threat. That combination produces false confidence, especially when output quality looks high but analytic coverage is shallow.
Failure mechanism: The system reuses past logic, prioritises familiar signals, and fails to surface weak or novel indicators that require human judgement to interpret.
Impact: New intrusion patterns can slip past detection, coverage gaps can remain hidden, and the organisation can accumulate rules that appear mature while missing the behaviours most likely to matter in a real incident.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK and OWASP API Security Top 10 address the attack and risk surface, while CIS Controls v8, NIST CSF 2.0 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | T1595 — Active Scanning | New threats are often found through behaviour patterns, not known indicators. |
| Recommendation — Map unfamiliar behaviours to ATT&CK techniques and build detections around observed tradecraft. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Detection engineering depends on trustworthy telemetry, retention and tuning of log sources. |
| Recommendation — Centralise and validate logs so analysts can confirm coverage before trusting detections. | ||
| NIST CSF 2.0 | DE.CM-01 — Monitoring for Anomalies and Events | The question is about whether detection processes still surface meaningful anomalies when AI does the work. |
| Recommendation — Continuously monitor for anomalies and review whether detections still reflect current behaviour. | ||
| OWASP ASVS | V16 — Security Logging and Error Handling | Detection logic quality depends on logging, validation and error handling in the detection workflow. |
| Recommendation — Instrument detection workflows so log quality and failures are visible during testing and tuning. | ||
| OWASP API Security Top 10 | API9 — Improper Inventory Management | New detections fail when teams lose track of which telemetry sources and analytic surfaces exist. |
| Recommendation — Maintain an inventory of telemetry and analytic surfaces before automating detection generation. | ||
Practitioner Guidance
What to verify: Require a human review step for any detection that targets a new tactic, new data source, or new environment, and confirm that the rule is backed by observable telemetry rather than inherited assumptions.
Decision rule: If the threat pattern is already well understood, AI can help draft and tune; if the pattern is novel or poorly documented, treat AI output as a starting point only and insist on analyst validation before production use.
Practitioner takeaway: AI can speed up detection engineering, but it cannot replace the human judgement that decides what deserves a detection in the first place.
Related resources from NHI Mgmt Group
- What breaks when security tools rely on legacy detection approaches for AI-driven social engineering?
- What breaks when AI is used for incident response or detection engineering without a realistic security harness?
- What breaks when application pentesting is left entirely to AI agents?
- What breaks when AI-generated internal tools are left running after a hackathon?
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 30, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org