How should teams evaluate machine learning models when classes are highly imbalanced?

Why This Matters for Security Teams

Highly imbalanced classes are common in security, fraud, and abuse detection, where the event you care about is rare but operationally critical. If a model is judged mainly by overall accuracy, it can look strong while missing the minority class almost entirely. That creates false confidence in triage, prioritisation, and automated decisioning, especially when the model is embedded in a broader control plane or workflow.

For security teams, the real issue is not whether a model is “good” in the abstract, but whether it is reliable under the failure mode that matters most. A model that catches most positives but floods analysts with false alerts can still be unusable. A model that is quiet and “accurate” may simply be ignoring the class that drives loss, risk, or harm. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the broader point that control effectiveness should be assessed against the outcome being protected, not a single headline metric.

In practice, many security teams encounter class imbalance only after a model has been accepted into production and starts missing the rare events it was intended to surface.

How It Works in Practice

The evaluation approach should start with the business cost of each error type, then map that cost to metrics and thresholds. For minority-class detection, precision shows how many flagged cases are truly positive, while recall shows how many true positives the model finds. F1 can help when precision and recall both matter, but it should not replace them, because a single score can hide where the model fails. Confusion matrices are essential because they show the actual distribution of true positives, false positives, true negatives, and false negatives.

Threshold tuning is usually where practical value appears. A model may rank cases well but still need an operating threshold that matches the team’s tolerance for missed detections versus alert volume. Precision-recall curves are often more informative than ROC curves when the positive class is rare, because they focus on performance where the class imbalance is most severe. Where decisions affect regulated or high-impact outcomes, teams should also document the intended use, the acceptable error profile, and the human review step. That is consistent with the risk-based approach in NIST AI Risk Management Framework.

• Use class-specific metrics, not only aggregate accuracy.
• Compare model output at multiple thresholds before selecting an operating point.
• Measure performance separately on validation, test, and post-deployment data.
• Track false positives and false negatives in the operational workflow, not just in the lab.
• Check whether reweighting, resampling, or synthetic data changes minority-class stability.

For AI systems with adversarial exposure, teams should also consider whether imbalance is being exploited through data manipulation or target suppression. MITRE ATLAS is useful for thinking about attack patterns that distort training or inference behaviour, while model governance should cover the provenance of labels and the integrity of the data pipeline. These controls tend to break down when the minority class is so small that validation data cannot support stable threshold selection because performance estimates become noisy and misleading.

Common Variations and Edge Cases

Tighter evaluation often increases analytical overhead, requiring organisations to balance statistical clarity against speed of deployment. That tradeoff becomes more visible when the minority class is extremely rare, when labels are delayed, or when ground truth is only partially available. In those cases, current guidance suggests using confidence intervals, time-sliced back-testing, or repeated resampling to avoid overreading unstable results. There is no universal standard for this yet, so the evaluation method should match the operational setting.

Some environments also need cost-sensitive evaluation rather than metric-first evaluation. For example, a fraud model may tolerate more false positives if manual review is cheap, while a safety or abuse model may prioritise recall to reduce missed detections. In agentic or automated workflows, the downstream risk is higher because a poor threshold can trigger tool use, escalation, or blocking actions without human context. That is why model evaluation should be paired with governance controls, especially where decisions affect customers, access, or financial loss. Where classes are highly imbalanced and the label quality is weak, even a well-chosen metric can mislead if the underlying data does not reflect the real operating environment.

Related resources from NHI Mgmt Group

• How should security teams govern machine learning models that may contain hidden backdoors?
• How should teams prevent bad data from reaching machine learning models?
• What do regulators expect from AI and machine learning risk models?
• How should teams govern AI workflows that span multiple machine learning platforms?