Zeek log-driven testing exposes regressions that PCAPs miss

At a glance

What this is: This is a practical guide to using Zeek logs as test input, with the key finding that log replay can uncover regressions and false negatives that do not show up in short PCAP-based tests.

Why it matters: It matters because security teams that rely on network analytics, detections, or parsing logic need a way to validate behaviour over long time windows, not just against synthetic snapshots.

Context

Traditional unit tests catch immediate logic errors, but they often miss bugs that emerge only after long-running state accumulates or unusual records appear in operational data. In network analysis and detection engineering, that gap becomes more visible when a rule or script behaves differently after weeks of normal traffic than it does in a short lab replay.

Detection engineering is the right mindset here because the problem is not just correctness in the moment, but repeatability across real telemetry and evolving edge cases. Zeek log replay turns historical data into a regression harness, which is a stronger fit for long-tail failure modes than a one-off packet capture.

Key questions

Q: How can security teams test detection logic against failures that only appear after long runtime?

A: Use historical log replay to drive the same code paths repeatedly, then compare behaviour across early and late runs. This exposes state drift, rare field combinations, and cleanup failures that short packet captures often miss. The point is not just to reproduce one bug, but to prove the control behaves consistently over time.

Q: Why do replay-based tests sometimes reveal bugs that PCAP-based tests do not?

A: PCAPs replay a moment in traffic, while logs can replay the operational history that built up the failure. When a script depends on accumulated state, table cleanup, or unusual record shapes, the long-run behaviour matters more than a single captured flow. Replay tests expose that difference by exercising the same logic at scale.

Q: What should teams do when a security script only fails after weeks or months in production?

A: Treat the incident as a test coverage gap and reconstruct the failure using the most representative logs available. Then isolate the smallest record or sequence that reproduces the issue, add it to the regression set, and keep the case in the harness so the same failure cannot return unnoticed.

Q: How should teams govern time-dependent state in detection or parsing pipelines?

A: Assume that stateful security logic can fail when execution time, cleanup timing, or input pacing differs from production. Govern those components with explicit teardown checks, long-run replay, and validation against accumulated data so the control is assessed as a lifecycle, not a one-off function.

Technical breakdown

How Zeek’s Input framework turns logs into events

Zeek can ingest TSV-formatted logs through its Input framework and map each record into an event handler. In this pattern, a log line is not just data for a table lookup, it becomes a trigger that can call package logic such as connection lifecycle handlers. That matters because it lets engineers test the same code paths that live traffic would exercise, while keeping the input deterministic and replayable. The key architectural idea is event synthesis from stored telemetry, not packet reconstruction. Practical implication: use log replay when you need to validate parsing, state handling, or edge-case logic against specific historical records. See the Detection Engineering Guide.

Practical implication: drive test cases from log records when packet captures are too small or too transient to reproduce the failure.

Why longitudinal log replay finds failures that PCAPs miss

Longitudinal analysis works because it replays months of operational logs against the same code path, exposing bugs that only emerge after state has drifted over time. PCAP-based tests are good at reproducing a single network moment, but they are weaker at surfacing cumulative state, rare field combinations, and regressions that appear only after prolonged execution. In this model, the log file becomes a retrospective corpus for behavioural verification, which is especially useful when the failure is intermittent and the original traffic is no longer available. Practical implication: build a regression set from historical logs whenever a bug only appears after extended runtime. See the Log Management and Retention Guide.

Practical implication: preserve representative historical logs so you can re-run the same failure conditions after every script change.

Why time-dependent state creates hidden test failures

Time-dependent analytics can fail when the engine’s notion of time does not advance in the same way it does during live traffic processing. In the article’s example, the script relies heavily on table structures and cleanup behaviour that assume normal temporal progression, so replaying large volumes of data without traffic can stress state management in unusual ways. That creates a class of failures that are not simple syntax issues, but lifecycle problems in how state is maintained, removed, or rebalanced. Practical implication: validate whether your test harness preserves the temporal assumptions your detection or parsing logic depends on.

Practical implication: test long-running state cleanup separately from single-event correctness to catch failures that only appear under accumulated load.

Threat narrative

Attacker objective: The operational objective is to avoid undetected analytic failures by forcing the package to reveal bugs that ordinary PCAP tests would miss.

1. Entry occurs when historical log data is fed into the test harness instead of live traffic, allowing dormant edge cases to surface in the script’s execution path.
2. Credential or access abuse is not the issue here; the failure mode is state accumulation and event timing under repeated log-driven execution.
3. Impact appears as false negatives, regression blind spots, or even engine failure when large log sets stress the package’s internal data structures.

Breaches seen in the wild

• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Log replay is a stronger regression control than packet capture when failure depends on time and state. The article shows that some detection or parsing bugs only appear after weeks or months of accumulated behaviour, not in short synthetic tests. That is a governance problem as much as a testing problem, because confidence in the script is being built on an incomplete sample of its operating conditions. Teams should treat historical telemetry as part of their quality assurance baseline, not as an ad hoc debugging aid.

Long-tail failure modes are where operational analytics usually breaks first. The bug described here is not a logic typo, it is a state-management failure that emerges when execution patterns differ from live conditions. That makes the lesson broader than Zeek: any security control that depends on accumulated state, timers, or cleanup cycles needs validation against extended replay, not just isolated examples. Practitioners should assume rare data shapes will eventually matter in production.

Temporal drift: when a test harness stops reflecting how time advances in production, cleanup and rebalancing assumptions can fail silently. That concept matters for network telemetry, detection pipelines, and any rule set that stores growing state across sessions. The article’s important insight is that the problem is not only data volume, but the mismatch between a frozen replay environment and a live event engine. Practitioners should test whether their tooling depends on time progression even when they think they are only validating logic.

Detection engineering needs replayable evidence, not just reproducible packets. The article’s method turns historical logs into a practical assurance mechanism for scripts that would otherwise be hard to validate. That is especially relevant when teams tune analytics for edge cases, because the right evidence may already exist in stored telemetry rather than in a preserved PCAP. Practitioners should build test corpora from operational logs wherever false negatives are costly.

The identity angle is indirect but real: event-driven security tooling is itself a governed runtime. When detection or parsing scripts process logs over time, they behave like managed automation with their own state, inputs, and failure modes. That is why teams managing security automation or detection pipelines should treat script lifecycle, input validation, and teardown logic as operational controls, not just engineering details. Practitioners should govern telemetry-driven automation with the same discipline they apply to other security-adjacent runtime systems.

What this signals

Replayable telemetry is becoming a quality control requirement for detection pipelines. As teams rely more on long-lived analytics, they need evidence that scripts behave the same way after thousands of events as they do on the first run. The useful programme shift is to treat historical logs as governed test assets, not as archive material.

Temporal drift is the failure mode that matters here: a control can look correct in isolated testing and still break when event timing, cleanup, or state accumulation changes. That means validation has to move from single-case correctness to long-run behavioural assurance, especially for pipelines that hold memory across sessions.

Security teams should expect more pressure to prove that telemetry-driven automation is stable under replay, not just functional in theory. The practical response is to fold historical data, teardown validation, and regression cases into the same assurance process that already covers rule logic and detection coverage.

For practitioners

• Build replay tests from historical logs Convert representative conn logs into deterministic test inputs so edge cases can be re-run after every script change without waiting for live traffic to reproduce them.
• Preserve raw logs for regression analysis Keep uncompressed, representative telemetry sets that capture rare states, unusual field values, and long-duration behaviour that short PCAPs do not expose.
• Separate state cleanup from single-event validation Design tests that check both the correctness of one event and the behaviour of accumulated state across many events, especially where timers or tables are involved.
• Validate teardown paths under long-running replay Make sure queued events finish processing before shutdown, and verify that terminate or end-of-data handling does not hide failures in the final state transitions.

Key takeaways

• Log-driven testing exposes failures that only appear after a package has accumulated state over time, which is why short PCAPs can miss them.
• The article shows that historical logs can be used as regression material, making rare false negatives easier to reproduce and prevent.
• The core control lesson is to validate cleanup, timing, and state handling under long-run replay, not just against isolated events.

Related NHI Mgmt Group resources: Detection Engineering Guide · Log Management and Retention Guide · SIEM Guide

Key terms

• Input Framework: Zeek’s Input Framework is the mechanism for ingesting external data and turning it into tables or events inside a script. In this article, it acts as a replay engine for historical logs, letting testers drive real code paths from stored telemetry instead of live traffic.
• Longitudinal Analysis: Longitudinal analysis is repeated examination of the same system or dataset across extended time periods. In security testing, it helps expose bugs that only emerge after state has accumulated, making it useful for regression validation, telemetry replay, and long-duration behavioural checks.
• Regression Test: A repeatable test used to confirm that a previously discovered failure has not returned after a model, policy, data, or tool change. In AI governance, regression tests are crucial because the same system can pass one day and fail the next when context changes.
• Temporal Drift: Temporal drift is the change in an agent’s behaviour over time without a corresponding code change. It can be caused by memory updates, retrieval differences, altered tool schemas, or policy changes, and it makes reproduction and governance harder because the runtime context is moving.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.