Behavioral email security exposes the limits of custom detection rules

At a glance

What this is: This is an analysis of why custom email detection rules become brittle and how behavioral models change the operating burden.

Why it matters: It matters because IAM, NHI, and human identity teams all face the same governance problem when control logic depends on constant manual tuning instead of durable policy and context.

By the numbers:

• Only 38% have automated certificate lifecycle management in place.
• 61% rely on spreadsheets or manual tracking for machine identity management.

👉 Read Abnormal AI's analysis of why custom email detection rules no longer scale

Context

Custom detection rules are a form of control logic that only works while the environment stays close to the assumptions encoded into those rules. In email security, that means a rule can start as precise and end as operational debt as sender patterns, executive names, and attack techniques evolve faster than the rule set can be maintained.

The identity governance issue is not limited to email. When security controls depend on handcrafted logic, the programme inherits a maintenance burden that looks different across human identity, NHI, and autonomous systems, but produces the same result: brittle coverage, delayed response, and too much reliance on tribal knowledge.

For teams already dealing with secret sprawl, machine identity growth, and increasingly dynamic access patterns, the lesson is straightforward. Static detection logic is easy to understand and hard to sustain, which is why behavioural models are gaining traction as a governance approach rather than just a detection feature.

Key questions

Q: How should security teams reduce dependence on custom detection rules?

A: Security teams should reduce custom rule dependence by using controls that learn normal behaviour, surface contextual evidence, and support repeatable triage. Rules should remain a bounded exception, not the default operating model. The goal is to lower maintenance burden while preserving analyst trust in why an alert fired.

Q: When do custom detection rules become a governance problem?

A: Custom detection rules become a governance problem when they outlive the conditions they were written for and require continuous tuning to stay useful. At that point, they are no longer just detection logic. They are an operational liability that consumes attention, creates blind spots, and depends on specialist knowledge to remain effective.

Q: What should teams measure to know if behavioural detection is working?

A: Teams should measure whether the system reduces false positives, shortens investigation time, and still explains its decisions clearly. A good behavioural control makes triage faster without hiding the evidence. If analysts cannot validate outcomes or if response speed does not improve, the model is not delivering operational value.

Q: How can organisations keep email detection resilient as threats change?

A: Organisations can keep email detection resilient by reviewing control drift, limiting bespoke logic, and preferring context-aware detections that adapt as message patterns change. Resilience comes from controls that learn with the environment rather than from accumulating more one-off rules.

Technical breakdown

Why custom detection rules become brittle

Custom rules encode yesterday’s threat shape into fixed logic. That works only until the attacker, the business process, or the communication pattern changes. In email security, even small variations such as a new vendor format, a different reply-to address, or an executive name change can turn a precise rule into noise or a blind spot. The deeper issue is operational: every rule becomes a long-term maintenance obligation that must be tested, tuned, documented, and explained. Over time, the rule library itself becomes part of the attack surface because teams spend more effort preserving detection logic than responding to risk.

Practical implication: treat custom rules as temporary controls with an owner, review cycle, and retirement path.

Behavioral baselines and identity context

Behavioral detection works by learning normal patterns for users, vendors, and communication flows, then flagging deviations that matter. Instead of asking engineers to define every malicious pattern in advance, the system compares each message or action against a living baseline. That baseline can include sender history, tone, timing, reply-chain behaviour, and expected counterpart relationships. This is especially useful in environments where attack patterns mutate quickly, because the control is tied to context rather than a specific signature. The governance value is that the model shifts from exception-based rule upkeep to continuous interpretation of behaviour.

Practical implication: validate whether the control measures behaviour against real identities and relationships, not just message content.

Explainability is part of operational trust

Autonomous or semi-autonomous detection systems still need clear evidence because analysts must understand why something was flagged. Explainability is not a nice-to-have layer on top of detection. It is the control surface that lets a team trust the outcome, investigate quickly, and defend decisions internally. In practice, that means contextual evidence such as unusual tone, unfamiliar addresses, or suspicious link behaviour must be surfaced in the workflow, not buried in a model output. Without that evidence, behavioural security risks becoming another opaque system that shifts work rather than reducing it.

Practical implication: require evidence-rich detections that support triage, audit, and post-incident review.

Threat narrative

Attacker objective: The attacker aims to exploit predictable controls so that a malicious message reaches the target and triggers a high-value action before detection logic adapts.

1. Entry occurs through a message that looks legitimate enough to bypass static rule logic, often by changing a single variable such as sender identity, reply path, or request format.
2. Credential or trust abuse follows when the recipient is persuaded to act on the message, granting access, data, or payment based on a pattern the rule set did not anticipate.
3. Impact lands as business loss, fraud, or downstream compromise, while the SOC absorbs the cost of tuning, explaining, and maintaining the failed rule logic.

Breaches seen in the wild

• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.
• LiteLLM PyPI package breach — LiteLLM PyPI supply chain attack, credentials stolen from users.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Custom detection logic is a governance liability once the threat pattern changes faster than the control can be maintained. Rules are only as durable as the assumptions they encode, and email threats mutate continuously. The consequence is not just false positives, but a control estate that quietly drifts out of alignment with real attacker behaviour. Practitioners should treat rule-heavy detection as a lifecycle problem, not a one-time design choice.

Behavioral email security is best understood as a shift from encoded exceptions to continuously updated identity context. The useful unit is not the rule itself but the relationship between sender, recipient, timing, and message behaviour. That matters across human identity and NHI governance because both depend on knowing what normal looks like before deciding what is suspicious. The practitioner takeaway is to prioritise controls that preserve context rather than multiply signatures.

Manual rule maintenance creates the same scaling problem that machine identity programmes face when inventories are incomplete. As control count rises, ownership fragments, testing slows, and knowledge concentrates in a few specialists. That pattern is visible in broader identity operations as well, where visibility gaps and manual tracking turn scale into risk. The implication is that teams should design for durable governance, not just better detection content.

Explainability is the difference between useful behavioural detection and opaque automation. If analysts cannot see the evidence behind a detection, the system merely relocates the burden from writing rules to interpreting black-box outputs. Clear contextual evidence turns detection into something that can be triaged, audited, and defended. Practitioners should require transparency as part of operational control, not as a cosmetic feature.

Rule-based detection makes the enterprise dependent on human-paced maintenance loops that attackers do not respect. Detection logic ages in hours, while tuning, review, and deployment happen in human cycles. That mismatch is the real failure mode. Security leaders should recognise that the operational model itself, not just the rules, needs redesign if the team is to keep pace with adaptive attacks.

From our research:

• Only 38% have automated certificate lifecycle management in place, according to The Critical Gaps in Machine Identity Management report.
• 59% of companies face greater difficulties auditing machine identities, primarily due to lack of clear ownership and limited visibility.
• For a broader lifecycle lens, the NHI Lifecycle Management Guide shows why ownership and retirement discipline matter as much as initial provisioning.

What this signals

Custom logic does not scale cleanly across identity programmes, which is why behavioural control models are increasingly attractive. When security teams inherit hundreds of handcrafted exceptions, the programme drifts toward maintenance rather than governance. In parallel identity domains, that same pattern shows up as manual tracking, unclear ownership, and slow remediation, which is why organisations need controls that reduce dependence on specialist tuning.

Our view is that the real decision is not rule-based versus AI-assisted detection, but maintainable control versus fragile control. A system that still requires humans to constantly rewrite logic has not escaped the scaling problem. The implication for readers is to evaluate whether a detection approach can survive business change without turning into another manual queue.

With 61% of organisations still relying on spreadsheets or manual tracking for machine identity management, according to The Critical Gaps in Machine Identity Management report, control fragility is already a wider identity problem. That is the signal to align email detection, machine identity governance, and human workflow controls around durable ownership rather than ad hoc maintenance.

For practitioners

• Inventory rule dependencies and ownership Document every custom detection rule, its business purpose, and the person or team responsible for tuning it. Prioritise the rules that protect high-value workflows, then retire logic that no longer has measurable value or clear ownership.
• Set expiry dates for handwritten logic Treat rule creation as a temporary response, not a permanent control. Require review intervals, false-positive thresholds, and a removal decision so outdated patterns do not accumulate into hidden operational debt.
• Shift to behaviour-linked evidence Prefer detections that surface contextual evidence such as sender history, reply-chain anomalies, and unusual request timing. Analysts should be able to validate a decision quickly without reverse-engineering the rule that fired.
• Reduce reliance on tribal knowledge Move detection knowledge out of private query languages and individual Git workflows into shared documentation and repeatable operations. If only a few engineers can understand the control, it is not governable at scale.

Key takeaways

• Custom detection rules are effective only while the environment and attacker behaviour stay close to the assumptions encoded into them.
• Behavioural systems improve resilience when they replace brittle logic with contextual evidence that analysts can validate quickly.
• The governance win is not less human involvement overall, but less human time spent maintaining detection logic that should be learning instead.

Key terms

• Custom Detection Rule: A custom detection rule is handcrafted security logic that flags activity when specific conditions are met. In practice, it is a maintenance-heavy control that must be tuned as business processes and attacker techniques change, which means its value depends on constant review and ownership.
• Behavioral Baseline: A behavioral baseline is a profile of normal activity built from observed communication or system patterns. It allows security tools to detect unusual behaviour without requiring every malicious pattern to be encoded in advance, which makes it useful when threats mutate faster than rules can be written.
• Detection Drift: Detection drift is the gradual loss of alignment between a security control and the environment it is meant to protect. It happens when rules, models, or assumptions are not updated as users, vendors, or threat patterns change, causing blind spots, false positives, or wasted analyst effort.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Abnormal AI: Key Insights on behavioural email security and the limits of custom detection rules. Read the original.