Agent egress detection: are your false positives actually measured?

TL;DR: A 0.1% false-positive rate against clean traffic can be meaningless if detectors are never tested against hard negatives such as quoted prompt injections, example API keys, or benign logs that look malicious, according to PipeLab.

Editorial analysis by NHI Mgmt Group, based on content published by PipeLab: “Your benign set should look malicious”.

Key questions

Q: How should security teams test detections against benign traffic that looks malicious?

A: Use a hard-negative benchmark built from real operational content, not just clean samples.

Q: Why do false-positive rates look better in lab tests than in production?

A: Lab tests often use easy negatives that do not resemble the traffic seen in production.

Q: What are the signs that a detection rule is overfitting its test set?

A: A rule is probably overfitting when it scores extremely well on clean data but generates a flood of alerts on normal documents, logs, or agent output.

Practitioner guidance

• Build a hard-negative corpus from real operational sources Pull benign lookalikes from docs, runbooks, CI logs, scanner output, support tickets, and tool schemas, then label each sample against the policy the detector is meant to enforce.
• Split precision reporting by traffic class Report false-positive rates separately for clean traffic and hard negatives, and include the denominator so the metric reflects requests, sessions, URLs, chunks, or tool calls.
• Keep a private holdout set outside tuning Reserve a test slice the detector never sees during rule development so the team can spot overfitting before the control reaches production.

Bottom line: Detection quality depends on whether the benign benchmark includes attacker-like lookalikes, not just on how low the headline false-positive rate appears.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →