Why do zero-shot classifiers often underperform in security alert triage compared with trained domain models?

Why zero-shot scoring breaks down in security triage

Zero-shot classifiers often look plausible in security because the text is familiar, but triage is not a generic language task. The decision depends on how your environment labels normal activity, which alerts matter, and which benign patterns recur in practice. Without that learned context, the model tends to overread surface wording and miss the operational meaning of the signal.

A trained domain model has seen examples from the same telemetry, product stack, and analyst workflow. That matters because “suspicious” in one environment may be routine in another, and the alert text alone rarely contains enough evidence to resolve that difference. The classifier needs exposure to local history, not just a prompt that explains the label names.

Security alerts are also shaped by context that zero-shot systems do not infer reliably, such as asset criticality, authentication path, time of day, user role, prior detections, and known baselines. A rule that sounds correct in prose can become noisy or brittle when the model has no training signal to anchor it to your actual distribution.

Why local training data changes the ranking of alerts

In triage, the most useful model is usually one that learns from labeled alerts, incident outcomes, and analyst dispositions. That training lets the classifier learn which combinations of features tend to correlate with true positives, false positives, or low-value noise in a specific environment. The gain is not abstract accuracy, it is better prioritization of limited human attention.

Domain models also capture negative evidence better. In security, an alert may look severe in isolation but be common, repeated, or explained by a control failure that is already understood. A zero-shot model often cannot separate “possible attack” from “known noisy pattern” because it lacks the historical examples that teach that distinction.

This is why framing and prompt wording can dominate zero-shot output. If the prompt nudges the model toward one interpretation, the classifier may follow that language even when the underlying alert distribution would support a different triage outcome. Trained models reduce that sensitivity by grounding decisions in observed examples rather than in prompt phrasing alone.

What practitioners should optimize for instead

For security alert triage, the right objective is usually not “Can the model read the alert?” but “Can it reproduce analyst judgment on this environment’s data?” That means measuring performance against local labels, not against a generic benchmark. The more variable the alert source, the more important it is to validate on the exact products, severity schemes, and workflows you operate.

Zero-shot can still be useful as a bootstrap or as a fallback when labels are scarce, but it should be treated as an exploratory layer. Once you have enough review history, supervised fine-tuning, weak supervision, or hybrid scoring usually gives a more stable result because the model learns the environment’s recurring benign patterns and abuse patterns.

In practice, the strongest setups combine model output with analyst feedback loops. That lets the system improve from disputed cases, drift in telemetry, and new attack patterns instead of freezing the initial prompt as if it were policy.

Risk and Threat Considerations

When zero-shot systems are used for triage, the main risk is not just lower accuracy, it is misplaced confidence. A model that has never learned the local alert distribution can elevate noisy events, suppress meaningful ones, or behave inconsistently as prompt wording changes, which increases both analyst fatigue and missed-detection risk.

Failure mechanism: The classifier relies on semantic similarity rather than domain priors learned from the environment, so it cannot weight local baselines, recurring benign activity, or alert provenance the way a trained model can.

Impact: Teams may spend more time on false positives, miss true positives hidden in familiar-looking text, and make triage decisions that vary with prompt framing instead of operational evidence.

Practitioner Guidance

What to verify: Test the model against your own historical alerts, not only a generic validation set. Look for error patterns by source, severity, asset class, and analyst disposition, because those slices usually reveal whether the model has actually learned the environment.

Decision rule: If the alert source is noisy, high-volume, or tightly coupled to local business context, prefer a trained or fine-tuned model over pure zero-shot scoring. Use zero-shot only when you need a temporary bootstrap or when the classification task is truly broad and low-stakes.

What practitioners underestimate: Prompt quality is not a substitute for labeled history. In security triage, the model must learn what your team already knows about normality, escalation thresholds, and false-positive patterns, otherwise the ranking will stay unstable.

Practitioner takeaway: Treat zero-shot triage as a starting point, not a production decision engine; the closer the alerting environment is to your own history and analyst practice, the more training data matters.

Related resources from NHI Mgmt Group

• Why do faster AI models often underperform on security investigation tasks?
• Why do security operations teams struggle to scale alert triage with traditional MDR models?
• Why do AI security controls often fail to transfer across deployment models?
• How should security teams use ZTNA context in cloud alert triage?