ML Model Evaluation and Monitoring for Security Teams

Security teams are asked to buy, approve and run machine learning detectors on the strength of a single headline number, and that number rarely survives contact with their own traffic. A malware classifier quoted at 0.99 F1, a phishing model with "99% accuracy" or an anomaly detector with a high ROC AUC can still bury a SOC in false positives once the real base rate and the passage of time are applied. This guide explains the metrics that matter for detection (precision, recall, F1, ROC AUC, average precision, calibration, MAE and RMSE), the test-set hygiene that keeps results honest, why public benchmarks saturate and get gamed, and how to monitor a model for drift once it is in production. It is written for detection engineers, SOC leads, AI governance owners and anyone judging an AI detection tool in a proof of concept. It sits beside the AI Security Evaluations Guide, which covers attack test suites for LLM applications and agents, and the AI Model Risk Management Guide, which covers validation governance. Sources were checked on 30 October 2026.

Key takeaways

Accuracy is the wrong headline for detection. When attacks are rare, a model that flags nothing scores high accuracy; ask for precision, recall and average precision at your own base rate.
Thresholds, not models, set the SOC workload. ROC AUC summarises every threshold at once; you run at one threshold, so judge the confusion matrix at the operating point you will actually use.
Most impressive results are leakage or time travel. Random splits that mix future samples into training inflate scores; peer-reviewed work on malware classifiers shows published F1 up to 0.99 falling under realistic time-aware splits.
Public benchmarks saturate and contaminate. Models reach strong scores quickly yet still fail simple challenge examples, and benchmark items can leak into training data, so treat leaderboard positions as weak evidence.
A model starts decaying the day it ships. Monitor input drift, score distributions, alert volumes and analyst verdicts, and set retraining and rollback triggers before go-live, as NIST AI RMF MEASURE 2.4 and MANAGE 4.1 expect.

The metrics that matter for detection

Every binary detector produces four counts against labelled data: true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). In scikit-learn, confusion_matrix returns them with true classes as rows, and the other classification metrics are arithmetic on those counts. The table gives the definitions as scikit-learn documents them and what each tells a security team.

Metric	Definition	What it answers in a SOC	Where it misleads
Precision	TP / (TP + FP)	Of the alerts raised, how many were real? Drives analyst trust and triage cost.	Depends heavily on base rate; a vendor's precision on a 50/50 test set will not hold in your environment.
Recall (sensitivity, detection rate)	TP / (TP + FN)	Of the real attacks, how many did we catch?	Easy to push to 1.0 by alerting on everything; always quote with precision.
F1 and F-beta	F1 is the harmonic mean of precision and recall; F-beta weights recall beta times as much as precision	One number for a fixed threshold. F2 suits "do not miss" use cases, F0.5 suits auto-blocking.	Ignores true negatives and hides which side is weak; report the two components too.
Balanced accuracy	Mean of recall on each class	Accuracy corrected for imbalance; drops to chance (0.5 for binary) if the model simply predicts the majority class.	Still a single threshold summary.
Matthews correlation coefficient	Correlation between predicted and true labels using all four cells	A balanced single score that stays informative on skewed data.	Hard to explain to non-specialists.
ROC curve and ROC AUC	True positive rate against false positive rate across all thresholds; AUC is the area under it	Ranking quality: how well the model orders malicious above benign.	On rare-event data a tiny FPR can still mean thousands of false alerts; AUC says nothing about the chosen threshold.
Precision-recall curve and average precision (AP)	AP is the sum over thresholds of (recall step) times precision, without interpolation	Better summary than ROC AUC for rare attacks. A random model's AP equals the fraction of positives, which gives a clear baseline.	Trapezoidal PR AUC is overly optimistic, per scikit-learn; ask which calculation was used.
DET curve	False negative rate against false positive rate across thresholds	Compares detectors around the low-FPR region where SOCs operate.	Less familiar to buyers.
Brier score and log loss	Strictly proper scoring rules on predicted probabilities	Whether risk scores are trustworthy as probabilities.	A lower Brier score does not by itself mean better calibration; it mixes calibration and discrimination.
MAE and RMSE	Mean absolute error; root of the mean squared error (scikit-learn `mean_absolute_error`, `root_mean_squared_error`)	For regression outputs such as risk scores, time-to-compromise estimates or forecast alert volumes.	RMSE punishes large errors more than MAE; a few outliers can dominate it.

Base rate: the arithmetic every buyer should do

Precision is not a property of the model alone. Take a detector with 95% recall and a 1% false positive rate, applied to 1,000,000 events of which 100 are malicious. It catches 95 attacks and raises about 9,999 false alarms, so precision is under 1%. The same model on a vendor's balanced test set would look excellent. Before any proof of concept, estimate the real prevalence of the behaviour in your telemetry and recompute precision from the vendor's TPR and FPR. The Alert Fatigue Reduction Guide covers what that volume does to analysts.

Calibration: can you trust the score?

A classifier is well calibrated when, among samples scored around 0.8, roughly 80% are actually positive. Calibration matters whenever a score is used as a probability: risk-based alerting, automated blocking above a cut-off, or combining scores from several models. scikit-learn's calibration_curve and CalibrationDisplay plot reliability diagrams, and CalibratedClassifierCV fits a calibrator on data the base model did not see. Its documentation gives practical rules:

Sigmoid (Platt scaling): suits small calibration sets but assumes a symmetric distortion that may not hold for highly imbalanced data.
Isotonic: corrects any monotonic distortion but overfits on small data; use it with more than about 1,000 samples.
Temperature scaling: multiclass only, a single parameter, does not change the predicted class.
Ranking effect: calibration generally leaves ROC AUC unchanged, though isotonic can alter it by introducing ties; use sigmoid if ranking must be preserved exactly.
Disjoint data: when calibrating an already trained model, the calibration data must be separate from the training data.

Test-set hygiene: why good numbers are often wrong

scikit-learn defines data leakage as using information at training time that would not be available at prediction time, which inflates estimates and leads to poor results on novel data. Its own worked example selects 25 features from 10,000 random ones before splitting: accuracy reads about 0.76 on data that is pure noise, and returns to about 0.5 when selection is fitted on the training split only. Security data adds problems of its own. Arp and colleagues reviewed 30 papers from top-tier security venues over ten years and found evaluation pitfalls widespread, and the TESSERACT study of Android malware classifiers (129,000 applications over three years) showed that spatial bias (test data that does not reflect deployment) and temporal bias (training on samples from after the test period) inflate published results.

Hygiene rule	What goes wrong without it	What to ask a vendor or your data science team
Split before any preprocessing, fit transforms on training data only	Scalers, imputers, PCA or feature selection fitted on all data leak test information	Is preprocessing inside a pipeline that is refitted per fold?
Time-ordered splits: train on the past, test on the future	Random splits let the model learn tomorrow's malware families and campaign infrastructure	What are the training and test date ranges? Is there a gap between them?
Realistic class balance in the test set	Balanced test sets overstate precision by orders of magnitude	What was the malicious to benign ratio, and how does it compare with production?
Group-aware splits	Near-duplicate samples, the same host, user, family or campaign appear on both sides	Were samples deduplicated and grouped by family, tenant or host before splitting?
A locked holdout used once	Repeated tuning against the test set turns it into training data	How many times was the holdout scored before the reported figure?
Label quality review	Labels from a single AV engine or rule set teach the model to copy that engine, including its mistakes	Where did labels come from, and what is the measured label error rate?
Fixed random seeds and recorded versions	Results cannot be reproduced or compared fold by fold	Can you rerun the evaluation and get the same numbers?

NIST AI RMF MEASURE 2.1 expects test sets, metrics and TEVV tools to be documented, and MEASURE 2.5 expects the limits of generalisation beyond development conditions to be documented. A vendor that cannot describe its test set at this level has not met that bar.

Benchmark saturation, contamination and gaming

Public benchmarks are useful for comparing research ideas and weak for buying decisions. The Dynabench authors observed that contemporary models quickly achieve outstanding performance on benchmark tasks while still failing simple challenge examples and struggling in real-world settings, and proposed dynamic benchmarks where humans write examples that fool the current model. For large language models the problem is worse because training data is opaque. Deng and colleagues (NAACL 2024) found evidence that benchmark items had leaked into training: when an answer option of an MMLU question was hidden, ChatGPT and GPT-4 reproduced it exactly in 52% and 57% of cases.

Saturation: when most systems score near the ceiling, differences between them are within noise and the benchmark no longer separates good from adequate.
Contamination: if test items are on the public web, a model may have memorised them; score gains then measure recall, not capability.
Gaming: repeated submissions, tuning to the benchmark's quirks and selective reporting of the best run all lift the number without lifting real performance.
Static versus adaptive attackers: a fixed benchmark of evasive samples measures yesterday's attacker. NIST AI 100-2 E2025 treats evasion and poisoning as attacks against the model itself (ATLAS AML.T0015 Evade AI Model, AML.T0020 Poison Training Data), so robustness has to be tested with adapted inputs. See the Adversarial Examples and Evasion Attacks Guide.

The practical response is a private, time-stamped evaluation set built from your own incidents, red team output and benign traffic, refreshed each quarter and never shared with vendors before they are scored.

Judging an AI detection tool in a proof of concept

A proof of concept should reproduce production conditions, not the vendor's demo. The ordered approach below works for EDR or NDR models, phishing classifiers, UEBA, fraud scoring and LLM-based triage.

Define the decision. Write down what the model output will trigger (an alert, a block, a ticket priority) and the maximum false alerts per day the team can absorb.
Build the evaluation set first. Use labelled data from a period after the vendor's training cut-off, at your real class balance, including known hard cases and recent incidents.
Run in shadow or audit mode. Score live traffic without acting on it, alongside the existing controls, for at least several weeks so weekly cycles are covered.
Fix the operating threshold. Choose it on a tuning slice, then report precision, recall and alerts per day on a separate slice at that threshold only.
Compare with a baseline. Score the incumbent rules or a simple model on the same data. A complex model that beats nothing has proved nothing.
Test robustness. Include obfuscated, repackaged or paraphrased variants of known-bad samples, and check whether small benign changes flip verdicts.
Measure explanations and workflow. Time how long analysts take to verify an alert and whether the stated reasons are correct.
Agree rollback. Before enforcement, document how to return to audit mode or the previous model version in minutes, and who can authorise it.

Vendor claim	What it often means	Evidence to request
"99% accuracy"	Measured on a balanced or majority-class-heavy set	Confusion matrix at the operating threshold and the class ratio of the test set
"AUC of 0.98"	Good ranking, threshold unspecified	Precision and FPR at the threshold you would run, plus the PR curve and AP
"Top of benchmark X"	Possibly saturated or contaminated benchmark	Results on your private holdout; benchmark release date versus training cut-off
"Learns continuously"	Model changes in production, possibly from attacker-influenced data	Change log, version pinning, retraining data controls and rollback mechanism
"Near zero false positives"	Counted only for alerts above a high-severity cut-off	False positives across all severities and the volume suppressed below the cut-off
"Risk score of 0 to 100"	Uncalibrated ranking presented as a probability	Reliability diagram on your data

For scoring vendors consistently, use the Buyer's Guide Methodology and Scoring Template, and for SOC tooling see the AI SOC Platform Buyer's Guide.

Decision guidance: which metric to gate on

Alert-only detectors: gate on recall at a fixed alerts-per-day budget; precision will be low and that is acceptable if triage is cheap.
Automated blocking or account lockout: gate on precision (or F0.5) and on calibration, because every false positive is an outage for a user or service.
Prioritisation and risk scoring: gate on ranking quality (AP, precision in the top k) and calibration, since analysts work from the top of the queue.
Small teams: prefer fewer, high-precision models with vendor-managed retraining, and insist on a written drift and rollback commitment in the contract.
Large teams with data science support: keep your own holdout, re-score vendor model updates before accepting them, and track metrics per business unit or tenant.

Drift and production monitoring

Lu and colleagues describe concept drift as unforeseeable changes in the underlying distribution of streaming data over time, and treat detection, understanding and adaptation as separate problems. Security data drifts faster than most because adversaries change behaviour in response to the detector, and because benign change (new software, new SaaS, a merger) shifts inputs too. TESSERACT showed classifier performance decaying over time and evaluated mitigations such as active learning. NIST AI RMF MEASURE 2.4 expects functionality and behaviour to be monitored in production, and MANAGE 4.1 expects post-deployment monitoring plans with user input, incident response and recovery.

Signal	What it detects	How to collect it
Input feature distributions	Data drift: new file types, new log sources, schema changes, broken parsers	Compare rolling windows with the training baseline using a distribution distance or two-sample test per key feature
Score distribution	Shift in model confidence even before labels arrive	Daily histogram of scores and share above the threshold
Alert volume per day and per source	Silent failure (volume collapses) or flood (volume spikes)	SIEM metric with alerting on deviation from a seasonal baseline
Analyst verdicts	Concept drift: precision falling as attackers or benign behaviour change	Closed-alert dispositions (true positive, false positive, benign true positive) fed back as labels
Missed-detection review	Recall loss, visible only from incidents found another way	Post-incident check of whether the model scored the relevant events
Model and data versions	Unannounced vendor updates or retraining on poisoned data	Version identifier logged with every score; change records

Set triggers in advance: for example, retrain or recalibrate when weekly precision on reviewed alerts falls a set amount below the accepted baseline, and return to audit mode when alert volume deviates beyond an agreed band. Retraining on production feedback is itself an attack surface (ATLAS AML.T0031 Erode AI Model Integrity covers degrading a model over time), so review new training data before use. The AI Data Poisoning and Backdoors Guide covers those controls.

Detection and measurement: KPIs that show it is working

Precision on reviewed alerts: weekly, per model and per source, against the baseline agreed at go-live.
Recall on known incidents: share of confirmed incidents in the period that the model scored above threshold.
Alerts per analyst per shift: the workload the model creates, compared with the budget set in the proof of concept.
Time from drift alarm to decision: how long it takes to recalibrate, retrain or roll back once a trigger fires.
Holdout freshness: age of the newest labelled samples in the private evaluation set.
Calibration error on recent data: gap between predicted and observed positive rates in the reliability diagram bins.

Where identity fits

Many security models score identities: login risk, impossible travel, service account anomalies and agent behaviour. Non-human identities are especially prone to drift because a deployment or new integration legitimately changes their behaviour overnight, which is a false positive generator if baselines are not refreshed. The pipelines that train and serve models also hold their own credentials (storage access keys, registry tokens, inference API keys), and a stolen training-data write credential is a poisoning path. See the AI Agent Behaviour Baselining and Anomaly Detection Guide and the ITDR Guide.

Framework mapping

Activity	NIST AI RMF 1.0	NIST SP 800-53 Rev. 5	NIST CSF 2.0
Document test sets, metrics and tools	MEASURE 2.1	SA-11 Developer Testing and Evaluation	ID.IM-02
Test under deployment-like conditions	MEASURE 2.3, MEASURE 2.5	SA-11	ID.IM-02
Evaluate security and resilience (evasion, poisoning)	MEASURE 2.7	RA-3 Risk Assessment	ID.IM-02
Check that the measurement method itself works	MEASURE 2.13	CA-7 Continuous Monitoring	ID.IM-02
Monitor in production and track emerging risk	MEASURE 2.4, MEASURE 3.1, MANAGE 4.1	CA-7, SI-4 System Monitoring	DE.AE-02
Assess vendor models over the relationship	MANAGE 2.2	RA-3	GV.SC-07

For the full framework see the NIST AI RMF Guide.

Common failure patterns

Buying on accuracy: a model approved on a balanced test set floods the queue at the real base rate, and analysts start closing its alerts unread.
Random splits on time-ordered data: evaluation mixes future campaigns into training, so the proof of concept looks far better than week one in production.
Tuning on the holdout: the team adjusts thresholds and features against the same test set until it is effectively training data.
Labels copied from one engine: the model reproduces an existing tool's blind spots and is then credited with "agreement".
No owner for drift: nobody watches alert volume, so a parser change silently drops a log source and the model scores nothing for weeks.
Accepting vendor model updates blind: a new version changes the score distribution and the old threshold no longer means the same thing.

Practitioner checklist

Write down the decision each model output drives and the daily false-alert budget.
Recompute vendor precision at your measured base rate before any proof of concept.
Build and lock a private, time-stamped holdout at real class balance, refreshed quarterly.
Require time-ordered, deduplicated, group-aware splits and preprocessing inside a pipeline.
Report precision, recall and alerts per day at one fixed operating threshold, plus the PR curve and AP.
Check calibration with a reliability diagram before scores drive automated actions.
Run new models and vendor updates in shadow mode against a baseline before enforcement.
Log model version with every score and feed analyst verdicts back as labels.
Set drift, recalibration and rollback triggers before go-live and name their owner.
Treat leaderboard and benchmark results as weak evidence until reproduced on your data.

ML model evaluation and monitoring for security teams: how to test vendor claims and catch decay