Scheduled baselines for network egress detection need durable state

At a glance

What this is: This is an analysis of how to build per-asset egress baselines for exfiltration detection without making the detector’s in-memory state carry the full learning horizon.

Why it matters: It matters because security teams need detection that adapts to asset-specific traffic patterns without sacrificing scale, restart resilience, or operational clarity in broader monitoring and identity-adjacent telemetry.

👉 Read TENZIR's analysis of durable baselines for network egress detection

Context

Long-horizon anomaly detection fails when the learning window is forced to live inside the streaming detector. That creates a control problem as much as an engineering problem, because the detector becomes responsible for both immediate judgment and historical memory. The article’s primary concern is network egress baselining, but the broader lesson is about separating stateful learning from live decision-making.

For IAM and NHI practitioners, the same pattern appears anywhere a control depends on per-entity history. A service account, workload, or application does not behave like a generic endpoint, so a one-size threshold misses meaningful deviations. The governance challenge is to preserve entity-specific context without making the runtime control brittle or restart-sensitive.

Key questions

Q: How should security teams implement long-horizon anomaly detection without bloating streaming state?

A: Keep the live detector narrow and push historical learning into a scheduled job that reads durable telemetry, computes per-entity baselines, and writes the result to a lookup table or materialized view. That separation preserves scale, makes restarts harmless, and keeps the runtime control focused on judging current behaviour rather than storing weeks of history.

Q: Why do entity-specific baselines work better than global thresholds for egress monitoring?

A: Because normal behaviour varies sharply by asset, account, and workload. A global threshold either misses quiet entities or floods you with noise from high-volume ones. Per-entity baselines let the detector compare each observation to the right reference point, which is the only way to make deviation scores operationally meaningful.

Q: What breaks when training data includes malicious behaviour?

A: The baseline can absorb the attack path as normal, which weakens or removes the deviation signal entirely. That is why training windows must exclude compromised periods, and why freshness, dispersion checks, and change review matter. If the control learns from poisoned history, it stops being a detector and becomes an amplifier of attacker behaviour.

Q: Who should own the baseline refresh logic in a detection pipeline?

A: The team that owns detection operations should own the refresh contract, even if another system computes the baseline. Clear ownership matters because the detector needs a dependable freshness signal, defined failure handling, and a stable enrichment schema. Without that governance, the control becomes hard to audit and easy to break during maintenance.

Technical breakdown

Why rolling thresholds fail at scale

A rolling detector improves on a fixed global threshold because it learns recent behaviour for each asset, but it also stores that history in memory. As the learning horizon grows from minutes to weeks, state grows with both the number of assets and the number of buckets retained. That makes the detector itself the storage layer, which is the wrong boundary for long-lived baselines. Restarting the process also resets learning, so the control repeatedly re-enters warm-up and loses continuity.

Practical implication: keep live detection windows short and move long-horizon history into durable storage with explicit refresh logic.

How per-asset baselines turn history into a control

The article uses a daily learner to reduce stored network activity into per-asset median and median absolute deviation values, then publishes those values into a lookup table. The streaming detector only enriches each event with the latest row and scores deviation against the asset’s own history. This is a clean separation of concerns: one pipeline learns from retained data, another judges current traffic, and neither has to own the other’s state. It is especially useful when assets have very different normal volumes.

Why baseline freshness and poisoning need explicit guards

A baseline is not static knowledge. If the learner stops, the updated_at value becomes stale and the detector should treat that as missing or unreliable context. The article also points out that seasonality can distort one-number baselines and that an attacker can poison the learning window by behaving like normal long enough to be absorbed into the model. Median and MAD make poisoning harder than a mean-based approach, but they do not eliminate the need to exclude suspect periods from training data.

Threat narrative

Attacker objective: The attacker wants to move data out quietly while making the detector learn that malicious traffic is ordinary.

1. Entry occurs through sustained outbound activity that blends into the asset’s normal traffic envelope rather than through a single obvious spike.
2. Escalation happens when the attacker remains active long enough for the learning window to incorporate malicious behaviour into the baseline.
3. Impact is achieved when exfiltration no longer stands out against the asset’s updated normal, reducing detection confidence and delaying response.

NHI Mgmt Group analysis

Long-horizon detection is a state-governance problem, not just a thresholding problem. Once a detector has to remember weeks of entity behaviour, it starts behaving like a data store with alert logic attached. That creates fragility around restarts, refresh cadence, and state ownership. For practitioners, the right question is whether the control owns history cleanly or smuggles it into runtime memory.

Per-asset baselining is a stronger governance model than fleet-wide normality. Network activity, service-account usage, and workload behaviour are all entity-specific, so a single threshold hides the differences that matter. In identity-adjacent programmes, that same logic applies to NHIs: each workload or service account has its own normal operating envelope, and policy should reflect that variance. The practitioner conclusion is to govern by entity context, not population average.

Baseline poisoning is the statistical version of standing privilege drift. If malicious behaviour is allowed to remain in the learning window, the control quietly normalises the attack path. That is structurally similar to access models that let excessive privilege persist until it becomes accepted practice. The named concept here is learning-window poisoning: history that includes hostile behaviour becomes the new definition of normal, and detection loses its edge. Practitioners should treat training data as a governed asset.

Separating learning from judging makes the detection contract auditable. The detector should score what it sees now, while a separate job owns the refresh of the baseline. That separation aligns better with NIST Cybersecurity Framework 2.0 and NIST SP 800-53 Rev 5 because it creates clearer accountability for monitoring, configuration, and system integrity. For teams operating NHIs or service accounts, the practical outcome is cleaner control ownership and fewer hidden dependencies in live detection.

Identity-aware telemetry will matter more as environments become more entity-driven. The same architectural pattern can support workload identity, service accounts, and other non-human entities because it preserves per-entity context without bloating runtime memory. That is where NHIMG sees the intersection with identity governance: the control plane is increasingly about governed entity behaviour, not just authentication events. Practitioners should design for entity-specific baselines now, before scale forces an unsafe simplification.

What this signals

Entity-specific telemetry will matter more as security teams move from coarse thresholds to governed behavioural baselines. The practical shift is toward controls that understand assets, service accounts, and workloads as distinct entities with different normal patterns, which is closer to how identity governance already thinks about access and responsibility.

Learning-window poisoning: once attackers can influence what the detector treats as normal, the control starts losing detection value even if the pipeline still runs. Teams should therefore treat historical telemetry, baseline refresh cadence, and excluded windows as governed inputs rather than passive data.

For identity-adjacent programmes, this pattern reinforces the value of the NHI Lifecycle Management Guide when behaviour is tied to workloads or service accounts. It also aligns with the NIST Cybersecurity Framework 2.0 view that monitoring is only useful when the underlying control state is reliable.

For practitioners

• Separate learning from detection Run scheduled baseline computation in a durable store and keep the streaming detector limited to live scoring and enrichment. This avoids coupling detection capacity to historical retention.
• Baseline by asset, account, or workload Group behaviour by the specific entity you are judging, not by the whole environment. A workstation, backup server, service account, and API consumer can all have different normal egress patterns.
• Add freshness checks to baseline use Refuse or downgrade baselines when updated_at is stale, so a silent learner failure becomes visible instead of quietly degrading detection quality.
• Exclude suspect periods from training data Remove windows that already generated findings or show signs of compromise before recomputing baselines, otherwise the model may absorb attacker activity as normal.
• Plan for restart-safe state ownership Make the lookup table or materialized view the durable contract, so a detector restart does not reset months of learned history or trigger repeated warm-up periods.

Key takeaways

• Long-horizon anomaly detection fails when state is forced into the live stream, because runtime memory cannot safely double as historical storage.
• Per-asset baselines improve detection quality by comparing each observation to the right normal, not to an environment-wide average.
• Detection pipelines need freshness, poisoning resistance, and restart-safe state ownership or they will quietly lose their value over time.

Key terms

• Per-asset Baseline: A per-asset baseline is a learned view of what normal behaviour looks like for one specific entity, such as a workstation, server, service account, or workload. It uses historical observations to compare current activity against that entity’s own typical range rather than an environment-wide average.
• Learning Window: A learning window is the historical period used to compute normal behaviour for a detector or model. Its length determines how much context the system retains, but if the window is too short it misses drift, and if it is managed poorly it can absorb attacker behaviour into the baseline.
• Median Absolute Deviation: Median absolute deviation, or MAD, is a robust measure of spread that resists outliers better than a standard deviation based on the mean. In detection pipelines, it helps keep a few extreme events from defining normal, which is useful when telemetry contains noise or early compromise signals.
• Baseline Poisoning: Baseline poisoning happens when malicious activity enters the training data and changes what the detector believes is normal. The result is a weakened or blind control, because future attack behaviour no longer stands out against the corrupted baseline.

What's in the full article

TENZIR's full article covers the operational detail this post intentionally leaves for the source:

• Runnable TQL examples for daily per-asset baseline calculation and lookup-table publication
• ClickHouse-backed query patterns for retaining weeks of network activity without expanding detector memory
• Alternative baseline placements, including materialized views and hybrid lake plus detector models
• Handling null baselines, freshness checks, and seasonality in the scoring pipeline

👉 The full TENZIR article covers the TQL pipeline design, ClickHouse options, and scoring logic in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to broader operational security decisions.