Novelty testing

The practice of evaluating an AI system against unusual, rare, or adversarial scenarios rather than only routine cases. In SOC settings, it checks whether the model remains useful when telemetry is incomplete, attack patterns are unfamiliar, or inputs are crafted to mislead it.

Expanded Definition

Novelty testing is a structured way to measure how an AI system behaves when the input no longer resembles the conditions used for training, tuning, or routine validation. In cybersecurity, that means probing model responses to unfamiliar telemetry, sparse logs, unusual attack chains, malformed prompts, or deliberately deceptive content. The goal is not only to find outright failures, but to understand whether the system degrades gracefully, rejects unsafe assumptions, or produces confident but unreliable output.

Definitions vary across vendors, because novelty testing is sometimes treated as a subset of red teaming, out-of-distribution testing, or adversarial evaluation. For NHIMG, the useful distinction is that novelty testing focuses on uncommon operational conditions, while red teaming often focuses on malicious intent. A mature program usually combines both, and aligns the results to governance controls described in the NIST Cybersecurity Framework 2.0 and the OWASP Top 10 for Large Language Model Applications where applicable.

The most common misapplication is treating novelty testing as a one-time adversarial prompt exercise, which occurs when teams validate only obvious edge cases and ignore operationally rare but security-relevant scenarios.

Examples and Use Cases

Implementing novelty testing rigorously often introduces uncertainty in evaluation results, requiring organisations to weigh broader coverage against the cost of building representative rare-case scenarios.

• Testing a SOC copilot with partial telemetry, where key fields are missing and the model must avoid overconfident attribution or fabricated conclusions.
• Feeding a detection assistant with a previously unseen attack sequence to see whether it generalises safely or incorrectly maps the pattern to a known benign event.
• Evaluating an AI analyst against malformed or contradictory prompts to determine whether it refuses unsafe instruction rather than improvising an answer.
• Checking a triage model against rare but legitimate incidents, such as simultaneous alert storms and infrastructure outages, to confirm it does not collapse useful prioritisation.
• Applying adversarially crafted inputs inspired by guidance from OWASP and the broader AI risk guidance in NIST AI Risk Management Framework to see whether the system remains bounded under stress.

Why It Matters for Security Teams

Security teams need novelty testing because AI systems often look dependable until the environment changes. In a SOC, that failure mode can mean a model performs well on routine alert clustering, then misclassifies a new intrusion path, misses an emerging campaign, or produces misleading summaries when logs are incomplete. That is especially important for agentic AI, where the system may have tool access and execution authority, making a bad inference operationally expensive instead of merely inaccurate.

Novelty testing also supports identity and access governance when AI systems rely on NHI-backed APIs, service accounts, or privileged workflows. If the model is expected to decide, recommend, or route actions involving secrets or elevated access, rare-case behavior must be tested before it can affect production controls. The practical question is whether the AI fails safely when its assumptions are wrong, not whether it performs well in the average case.

Organisations typically encounter the real cost only after an unfamiliar event exposes brittle model behavior, at which point novelty testing becomes operationally unavoidable to explain the failure and contain its impact.

Related resources from NHI Mgmt Group

• When does static testing create a false sense of security?
• Why do application testing tools matter for NHI governance?
• Should organisations invest in AI offensive testing before adversaries do?
• What is the difference between compliance testing and identity recovery testing?