Novelty Detection

What novelty detection does

Novelty detection learns a baseline of normality and then flags observations that fall outside it. That makes it useful when the important problem is not classifying known bad patterns, but noticing something the system has not seen before.

In security operations, that distinction matters because many attacks, misuses, and failures begin as unfamiliar behaviour rather than a match to a known signature. Novelty detection is therefore often paired with anomaly scoring, threat hunting, or triage logic that decides whether an unexpected event deserves investigation.

Where novelty detection is used

Novelty detection appears in intrusion detection, malware screening, fraud analytics, and monitoring for unusual user, host, or network behaviour. It is also used in machine learning pipelines to spot out-of-distribution inputs, bad sensor readings, corrupted data feeds, and other conditions that do not fit the trained model.

Because the model is built from historical examples, it works best when the notion of “normal” is reasonably stable. When the environment changes frequently, the system may confuse legitimate change with novelty, which is why tuning, retraining, and context-aware thresholds are part of effective use.

Why novelty detection is hard

The core challenge is that “different from normal” is not the same as “dangerous.” A novel event may be benign, malicious, or simply rare. Systems that overreact create alert fatigue, while systems that are too permissive miss early warning signs and allow unusual activity to blend into the background.

It is also possible for adversaries to exploit the uncertainty around novel behaviour. If defenders rely too heavily on static rules, a new technique, altered payload, or unusual sequence of actions may evade controls long enough to matter. MITRE ATT&CK Enterprise Matrix is a useful companion for mapping those behaviours to known adversary techniques, while SANS Security Resources provides practitioner-oriented material on detection engineering and incident response.

How practitioners should interpret alerts

Novelty detection should be treated as a prioritisation signal, not a verdict. The strongest operational value comes when an alert is correlated with asset criticality, recent changes, expected business processes, and other telemetry that can tell the difference between a harmless first-time event and a genuine deviation worth action.

When teams need a countermeasure lens rather than a pure detection lens, MITRE D3FEND helps connect detection outcomes to defensive actions. That makes it easier to decide whether the right response is suppression, tuning, enrichment, containment, or deeper investigation.

Risk and Threat Considerations

Novelty detection can fail in two opposite ways: it can miss the first sign of a new attack pattern, or it can generate so many false positives that analysts stop trusting the output. The risk is highest when the model has weak training data, limited context, or no feedback loop to separate benign change from suspicious deviation.

Failure mechanism: Attackers can deliberately vary tools, timing, payload shape, or operational sequence to appear unfamiliar without triggering known signatures, while environment drift can make legitimate activity look novel and desensitise defenders.

Impact: The result can be delayed detection, missed triage, wasted analyst effort, and a growing blind spot around low-and-slow compromise activity or unusual system behaviour.

Related resources from NHI Mgmt Group

• When should organizations prioritize the detection of shadow AI agents?
• What are effective practices for operationalizing NHI threat detection?
• How do organisations reduce false positives in secret detection pipelines?
• When does regex-based secret detection become too unreliable for production use?