Network Monitoring and Packet Capture Guide: Flow Data, Full Capture and Retention

When an incident is declared, the first network question is almost always "what talked to what, and when", and many organisations find they kept nothing that can answer it. Network evidence comes in three layers: flow records, protocol metadata and full packet capture. Each costs more to collect and store than the one before, and each answers different questions. This guide explains what NetFlow, IPFIX, cloud flow logs, Zeek and Suricata metadata, and pcap or pcapng capture give you, where to collect them, how to size storage and set retention, what encryption leaves visible, and how to handle the privacy and credential risks of holding raw traffic. It is for network and security engineers, SOC and incident response leads. Detection engines and rule tuning are covered in the IDS and IPS Guide, and behavioural analytics products in the NDR Guide.

Key takeaways

• Flow is the long-retention baseline: cheap enough to keep for months, enough to show who talked to whom, never enough to show what was said.
• Protocol metadata is the working layer for investigations: Zeek and Suricata logs record DNS names, HTTP requests, TLS server names and connection states at a fraction of packet storage.
• Full capture is a rolling buffer, not an archive: at 1 Gbps uncompressed it fills roughly 11 TB a day, so keep days of packets and longer of metadata.
• Encryption moves value from payload to metadata: TLS 1.3 and Encrypted Client Hello hide more of each session, which makes connection timing, volumes and peers more important.
• Captured traffic is sensitive data: it contains personal data and often cleartext credentials, so it needs access control, encryption, a retention limit and a lawful, proportionate purpose.

The three layers of network evidence

Most organisations need all three, at different scopes: flow everywhere, metadata at key boundaries, and full capture at a few high-value points.

Flow data: NetFlow, IPFIX and cloud flow logs

• Definition: RFC 3954 (Cisco NetFlow version 9, Informational, October 2004) defines a flow as a unidirectional sequence of packets with common properties passing a device. A TCP conversation is therefore normally two flow records.
• Templates: NetFlow v9 and IPFIX describe record layouts in templates (IDs 256 to 65535) that exporters resend periodically and collectors must store. A collector that restarts and has not yet seen the template cannot decode records, which shows up as gaps.
• IPFIX: RFC 7011 (Internet Standard 77, September 2013) is the IETF standard that grew out of NetFlow v9; its message version is 10. It can run over SCTP, UDP or TCP. Prefer it for new deployments where devices support it.
• Timeouts: RFC 3954 describes an inactive timeout (no packets seen) and an active timeout that exports long-lived flows at intervals. Long active timeouts make a multi-hour exfiltration session appear only when it ends, so set them short enough for detection.
• Sampling: exporters may sample packets (the NetFlow fields SAMPLING_INTERVAL and SAMPLING_ALGORITHM record how). Sampled flow is fine for capacity planning and poor for security, because short connections such as a single beacon can vanish entirely.
• Transport: NetFlow export usually runs over UDP, which is not congestion-aware; RFC 3954 recommends a dedicated link between exporter and collector where congestion matters. Monitor collector drop counters.
• Cloud flow logs: AWS VPC Flow Logs, for example, record IP traffic to and from network interfaces, are collected outside the traffic path so they do not affect throughput, and are delivered to CloudWatch Logs, S3 or Data Firehose. They contain no payload. Each provider documents traffic types it does not log, so read those limitations before relying on flow logs for a cloud investigation. See the Cloud Network Security Guide.

Protocol metadata with Zeek and Suricata

Zeek describes itself as a passive network traffic analyser. By default it writes structured logs (tab-separated or JSON) for every connection, HTTP sessions, DNS requests and replies, certificates and SMTP, and it can carve files from traffic for sandbox analysis. Suricata's EVE JSON output covers similar ground, with event types including flow, dns, http, tls, smb, ssh, krb5, rdp and fileinfo. Running both on one sensor is common: Zeek for rich logs and scripting, Suricata for signatures.

Zeek's conn.log is the record most investigations start from. Its conn_state field summarises how each connection went:

• Other key fields: uid joins a connection to its DNS, HTTP, TLS and file records; orig_bytes and resp_bytes show direction of transfer (a large originator byte count to an external host suggests upload, T1048); missed_bytes above zero means content gaps that usually break protocol analysis; history letters record flags per direction (uppercase originator, lowercase responder).
• Quality signal: a high share of OTH states or non-zero missed_bytes means the sensor is missing packets or one direction of traffic. Fix capture before tuning analytics.
• Join keys across tools: keep the same field naming across Zeek, Suricata and the SIEM, and enable a shared flow identifier where both tools support one, so analysts can pivot from an alert to the connection and its packets. See the SIEM Guide.

An illustrative Zeek fragment that writes a notice when an internal host makes cleartext FTP logins. Treat it as a starting point, not production code:

@load base/frameworks/notice
redef enum Notice::Type += { Cleartext_FTP_Login };
event ftp_request(c: connection, command: string, arg: string)
    {
    if ( command == "PASS" && Site::is_local_addr(c$id$orig_h) )
        NOTICE([$note=Cleartext_FTP_Login, $conn=c,
                $msg="FTP password sent in cleartext",
                $identifier=cat(c$id$orig_h, c$id$resp_h)]);
    }

Full packet capture: pcap, pcapng and capture tools

• Formats: the classic libpcap format stores packets with a timestamp and lengths. pcapng adds blocks: Section Header, Interface Description, Enhanced Packet, Simple Packet, Name Resolution, Interface Statistics, Decryption Secrets and Custom blocks. It records which interface each packet came from, per-packet comments, drop counts and hashes. pcapng is still an IETF Internet-Draft (draft-ietf-opsawg-pcapng, intended Informational), although tools have used it for years.
• Ad hoc capture with tcpdump: -i selects the interface, -w writes raw packets, -n turns off name resolution, and -s sets the snapshot length (default 262144 bytes, effectively whole packets). -Z drops root privileges after opening the device, which you should always use.
• Rolling buffers: -C starts a new file once the current one exceeds a size (in millions of bytes), -G rotates by time, and -W with -C creates a ring that overwrites the oldest file. -z runs a command such as gzip on each closed file. For example: tcpdump -i eth1 -n -Z pcapuser -C 1000 -W 200 -w /capture/edge.pcap keeps about 200 GB of the most recent traffic.
• Continuous capture platforms: tools such as Arkime write PCAP to local disk and session metadata to OpenSearch or Elasticsearch, delete the oldest PCAP automatically as disks fill, compress PCAP (zstd is the default in recent versions) and can encrypt stored PCAP.
• Triggered capture: where continuous capture is too expensive, capture on demand for a host or flow when an alert fires, and accept that you lose the packets from before the trigger.
• Analysis: keep a documented chain of custody (hash each file on closure) when captures may become evidence. See the Network Forensics Guide.

Where to collect

CISA and partner agencies' December 2024 Enhanced Visibility and Hardening Guidance for Communications Infrastructure recommends flow monitoring with exporters and collectors at key ingress and egress points, packet capture where appropriate with locations and retention chosen by need, and an out-of-band management network isolated from production. It was written for telecoms after the Salt Typhoon intrusions, but the pattern fits any enterprise.

• Taps versus SPAN: hardware taps copy every packet, including errors, and do not compete with switching; mirror ports are cheaper but can drop copies when oversubscribed. Use taps where captures may be evidence.
• Packet brokers: aggregate taps, remove duplicates, strip tunnels and send each tool only what it needs, so sensors are not overwhelmed.
• Segments define placement: collection points should match trust boundaries. See the Network Segmentation Guide, and for OT the OT Asset Visibility Guide.

Storage sizing and retention

Size storage from measured traffic, not link speed. Arkime's documentation gives the working figure: 1 Gbps of uncompressed traffic uses roughly 11 TB of disk per day (1 Gbps is 125 MB per second, multiplied by 86,400 seconds is 10.8 TB). Real links average well below line rate, compression helps, and excluding bulk encrypted sessions after their first packets can cut volume substantially.

• Tiered retention: a common pattern, which Arkime's own example follows, is packets for about a week and session metadata for about a month, with flow records kept for longer still. Set the periods from how long your incidents take to discover, legal hold needs and privacy limits.
• Baselines need history: CIS Controls v8.1 safeguard 8.10 asks for audit logs to be kept for at least 90 days. Flow and metadata kept for at least that long lets you compare current service-to-service traffic with last quarter.
• Retention must be enforced by the system: Arkime deletes PCAP as disks fill but does not expire metadata unless you configure index lifecycle management or a scheduled job. Check that each tier actually deletes on schedule.
• Protect the store: SP 800-53 AU-9 (protection of audit information) and AU-11 (retention) apply to network evidence as much as to logs. Keep a copy off the sensor, restrict deletion rights, and monitor for gaps.
• Log management practice: NIST SP 800-92 Revision 1 (initial public draft, October 2023) frames log management as generating, transmitting, storing, accessing and disposing of log data. See the NIST SP 800-92 Log Management Guide and the Log Management and Retention Guide.

Encryption: what monitoring still sees

TLS 1.3 (RFC 8446) encrypts every handshake message after the ServerHello, including the server certificate, and requires forward-secret key exchange, so a captured session cannot be decrypted later with the server's private key. RFC 9849 (Encrypted Client Hello, March 2026) also hides the real server name and ALPN in an encrypted inner ClientHello, leaving only the outer public name visible.

• Still available: addresses, ports, byte and packet counts, timing, connection state, the outer ClientHello and client fingerprints such as JA3 and JA4 where enabled. These are enough for beaconing, volume and new-peer detections.
• Reduced: server names (with ECH), certificate details, URIs, file contents and anything that needs the payload.
• Decryption options: session keys exported by endpoints you control can be stored with captures (pcapng has a Decryption Secrets block), or traffic can be decrypted at a proxy. Both create highly sensitive key material and need the same governance as the captures. See the TLS Inspection Guide.
• DNS as a substitute: resolver logs from your own resolvers often name the destination that ECH hides on the wire, as long as clients use them.

Privacy, credentials and access to captures

• Personal data: packets contain email content, browsing, names and identifiers. Under UK GDPR and EU GDPR, monitoring needs a lawful basis, transparency, data minimisation and usually a data protection impact assessment where monitoring is systematic. The ICO's guidance on monitoring workers was under review in 2026 following the Data (Use and Access) Act, so check the current text before relying on it.
• Minimise by design: collect flow and metadata widely, full capture narrowly; truncate payloads where headers are enough; exclude known high-volume, low-risk traffic.
• Credentials in captures: any capture of FTP, Telnet, HTTP Basic, SNMPv1/v2c, LDAP simple binds or TFTP configuration transfers contains reusable secrets. CISA's guidance is to disable Telnet, FTP, TFTP, SSH v1, unencrypted HTTP management and SNMP v1/v2c, and to use only SNMPv3 with authentication and encryption. Until that is done, treat capture stores as credential stores. See the Network Ports and Protocols Security Guide.
• Service-to-service baselines: machine identities have stable peers, ports and volumes. Flow and conn.log history let you list, per service account or workload, who it normally talks to, and alert on new peers or new protocols. See the Service Account Security Guide.
• Capture system access: capture appliances, sensors and query consoles use service accounts, API tokens and storage credentials. Scope them per system, keep them in a secrets manager, require MFA for analysts and log every search and export.
• Attackers capture too: network sniffing (T1040) and adversary-in-the-middle (T1557) are credential-theft techniques. Unexpected capture processes, promiscuous interfaces and new mirror sessions on switches deserve alerts.

How to implement

1. Write the questions: for example, "which internal host contacted this address in the last 90 days", "what was downloaded from this URL", "which workloads use cleartext protocols". Map each to a layer.
2. Turn on flow everywhere first: IPFIX or NetFlow from routers and firewalls without sampling where security matters, cloud flow logs on every VPC, collectors with drop monitoring.
3. Add metadata sensors at boundaries: Zeek or Suricata on taps at the internet edge and data centre core. Check OTH and missed_bytes rates in the first week to validate capture.
4. Pilot full capture at one point: measure real daily volume for two weeks before buying storage, then size for your retention target with headroom.
5. Complete the privacy work: DPIA, access model, retention schedule and an exclusion list, approved before wide rollout.
6. Integrate: send flow and metadata to the SIEM or data lake with asset and owner context, and link alerts to the relevant packets.
7. Test retrieval: run a quarterly exercise to pull flows, metadata and packets for a known event from 30, 60 and 89 days ago.
8. Roll back safely: passive collection can be removed without network impact; keep mirror and tap changes under change control so a misconfigured mirror session can be reverted.

Detection and measurement

• Coverage: percentage of internet egress points, zone boundaries and VPCs producing flow and metadata.
• Capture quality: sensor and collector drop rate, and the share of connections with OTH state or non-zero missed_bytes.
• Retention achieved: oldest available record per tier compared with the target, checked automatically every day.
• Retrieval time: minutes from request to having the packets or flows for a given host and time window.
• Cleartext exposure: number of internal services still using cleartext authentication, trending to zero.
• Baseline drift: new service-to-service peer pairs per week that were not approved.

Framework mapping

Common failure patterns

• No visibility of the management plane: in the Salt Typhoon telecom intrusions, Cisco Talos described device configurations exfiltrated over TFTP and FTP (T1602.002), SNMP, TACACS+ and RADIUS traffic captured to collect shared secrets, and a custom tool that captured packets across chains of devices while clearing logs. Flow and metadata from the management network would have shown the transfers and the new SSH services.
• Sampled flow used for security: 1-in-1000 sampling hides short connections and single beacons.
• Capture sized on paper: storage bought for link speed or an average, then retention silently shrinks to hours at peak.
• Retention nobody enforces: metadata indexes grow forever, or PCAP is deleted before an investigation reaches it.
• Captures left open: pcap files on shared drives or in tickets, full of personal data and cleartext credentials.
• One-sided capture: asymmetric routing means the sensor sees only one direction, and nobody checks the OTH rate.

Practitioner checklist

• Enable unsampled IPFIX or NetFlow at security boundaries and flow logs on every cloud network.
• Run Zeek or Suricata metadata logging at internet and data centre boundaries.
• Keep a rolling full-capture buffer at the internet edge and remote-access points, sized from measured traffic.
• Set and enforce retention per tier: packets for days, metadata for weeks, flow for at least 90 days.
• Hash and store captures with access control, encryption and logged access.
• Complete a DPIA and publish an internal monitoring notice before wide full capture.
• Find and remove cleartext authentication protocols seen in metadata, and rotate exposed credentials.
• Baseline service-to-service traffic and alert on new peers and protocols.
• Monitor drop rates, OTH share and oldest-record age daily.
• Rehearse retrieval of 30, 60 and 89 day old evidence each quarter.

Standards and references

• IETF: RFC 7011, Specification of the IP Flow Information Export (IPFIX) Protocol
• IETF: RFC 3954, Cisco Systems NetFlow Services Export Version 9
• IETF: PCAP Now Generic (pcapng) Capture File Format draft
• CISA: Enhanced Visibility and Hardening Guidance for Communications Infrastructure
• NIST: SP 800-92 Rev. 1, Cybersecurity Log Management Planning Guide (initial public draft)
• Zeek Project: Connection analysis script and conn.log fields
• The Tcpdump Group: tcpdump manual page
• Arkime: FAQ

Related NHI Mgmt Group resources: NDR Guide · IDS and IPS Guide · Network Forensics Guide · SIEM Guide · Network Segmentation Guide