What breaks when detection engineering is left entirely to AI?

What breaks is the ability to create useful detections for unfamiliar threats. AI can reuse known logic, but it struggles to decide which signals matter when the attack pattern is new or poorly documented. Without human analysts, the SOC risks encoding old assumptions into new workflows and missing the behaviours that actually matter.

Why This Matters for Security Teams

Detection engineering is not just rule-writing. It is the discipline of deciding what matters, what can be ignored, and how uncertainty should be handled when the environment changes. When AI is left to do this alone, it tends to optimise for pattern repetition instead of threat relevance. That can produce tidy-looking detections that map well to historical telemetry but fail against novel tradecraft, especially when an adversary intentionally avoids known indicators. Guidance from the NIST Cybersecurity Framework 2.0 still matters here because it places detection inside a broader governance and continuous improvement cycle, not as a standalone automation task.

The practical risk is that AI can scale inconsistency faster than it scales judgment. If the model is trained on past incidents, it may overvalue signals that were important in yesterday’s environment and underweight weak but meaningful anomalies in today’s one. That is especially dangerous in SOC operations where adversaries test timing, living-off-the-land behaviour, and low-and-slow activity designed to stay beneath threshold-based logic. In practice, many security teams discover this only after a gap in analyst oversight has already turned a miss into an incident.

How It Works in Practice

Effective detection engineering still depends on a human-led loop: define the threat hypothesis, select the telemetry, test for coverage, then tune based on false positives, false negatives, and operational context. AI can assist at each step, but it cannot reliably replace the decision about whether a signal is actually security-relevant. This is where models often help with correlation, summarisation, and draft rule generation, while analysts retain authority over prioritisation and validation.

In mature environments, AI works best when it is constrained by an explicit detection strategy. That usually means:

• using AI to cluster alerts and suggest related patterns, not to declare a detection complete;
• reviewing AI-generated rules against known attack techniques in MITRE ATT&CK or a comparable technique catalogue;
• keeping telemetry provenance visible so analysts can judge whether the data source is trustworthy, complete, and timely;
• testing detections against adversarial behaviour, including prompt injection or log manipulation where AI systems touch security workflows;
• requiring human approval for high-impact changes such as suppressions, thresholds, and escalation paths.

The core issue is not whether AI can generate detections. It can. The issue is whether it can distinguish an interesting outlier from an operational nuisance when context is missing or ambiguous. NIST’s guidance on AI risk management is useful here because it treats performance, robustness, and human oversight as linked concerns, not separate checkboxes. The best operational model is AI-assisted detection engineering with analyst-owned validation, feedback, and exception handling. These controls tend to break down when telemetry is sparse or heavily normalised across a cloud estate because the model has too little context to separate benign automation from true attacker activity.

Common Variations and Edge Cases

Tighter detection automation often reduces analyst workload, but it also increases the risk of blind spots, so organisations have to balance speed against epistemic confidence. That tradeoff becomes sharper in environments where the available data is noisy, fragmented, or over-abstracted by upstream platforms.

Best practice is evolving for AI-generated detections in three common edge cases. First, in highly dynamic cloud or DevOps environments, behaviour changes so quickly that a model may keep relearning yesterday’s baseline instead of recognising meaningful deviation. Second, in low-volume but high-value contexts such as privileged access monitoring, a small number of events may matter more than any obvious pattern, which makes pure statistical ranking unreliable. Third, in adversarial settings, attackers may deliberately poison training examples, manipulate logs, or stage activity to look normal, which means detection engineering must include validation of data quality and provenance.

There is also an important boundary between assistance and authority. AI can suggest detections for known techniques, but it should not be the only decision-maker for new campaigns, policy exceptions, or high-severity escalations. For that reason, many teams pair AI with CISA threat-informed prioritisation and manual threat hunting to keep the programme anchored to real adversary behaviour. Where security teams rely on automated tuning alone, detections often drift toward what is easy to measure rather than what is operationally meaningful.

Related resources from NHI Mgmt Group

• What breaks when AI-generated internal tools are left running after a hackathon?
• What breaks when service accounts and API keys are left unrotated in AI systems?
• What breaks when AI workflow approval is left informal?
• What breaks when runtime detection is the main control for AI agent security?