Full packet capture does not scale to every link, but without network records a SOC cannot see lateral movement, beaconing or bulk exfiltration that never touches an endpoint agent. Flow telemetry fills that gap: routers, switches, firewalls and cloud platforms summarise each conversation (who talked to whom, on which ports, how much, when) and export the summaries to a collector. NetFlow version 9 and its IETF successor IPFIX are the main standards, with sFlow and cloud flow logs as close relatives. This guide explains the formats, which fields matter for security, how to design collection, how to secure the export path, and the detections and metrics that make flow data worth keeping. It is for network security engineers, SOC and detection engineers. For packet capture and general network monitoring, see the Network Monitoring and Packet Capture Guide.
Key takeaways
- IPFIX is the standard; NetFlow v9 is its parent. RFC 7011 (IPFIX, STD 77, September 2013) is an Internet Standard with message version 10; RFC 3954 (NetFlow v9, October 2004) is an Informational description of Cisco's format.
- Flows are metadata, not content. A flow record holds addresses, ports, protocol, byte and packet counts, timestamps and TCP flags, which is enough for many detections and cheap to keep for months.
- Templates make the format flexible and fragile. Both formats describe records with templates; a collector that loses or never receives a template cannot decode the data.
- Placement and sampling decide what you see. Unsampled flows at internet edges, data centre cores and cloud VPCs give the most security value; heavy sampling hides small, slow flows such as beacons.
- The export path is an attack surface. Flow export usually runs over UDP without authentication, so isolate it on a management network or protect it with TLS or DTLS where supported.
Flow formats compared
| Format | Status and source | Structure | Notes for security teams |
|---|---|---|---|
| NetFlow v5 | Cisco legacy format | Fixed record layout | IPv4 only and no extensibility; still found on older devices |
| NetFlow v9 | RFC 3954, Informational, October 2004; not a candidate for Internet Standard | Template FlowSets (FlowSet ID 0) describe data records; Template IDs 256 to 65535; version field 9 | No fixed port in the RFC; UDP usual; templates expire at the collector unless refreshed |
| IPFIX | RFC 7011, Internet Standard (STD 77), September 2013, obsoletes RFC 5101 | Template Sets (Set ID 2), Options Template Sets (Set ID 3), Data Sets 256 and above; version field 10 | Information Elements registered by IANA; supports variable-length and enterprise-specific fields |
| sFlow | Industry specification (sflow.org) | Sampling of packets on switch and router interfaces | Designed for high-speed links; always sampled, so less suited to spotting single small flows |
| Cloud flow logs (for example AWS VPC Flow Logs) | Provider documentation | Provider-defined fields, for example the AWS version 2 default format | Aggregated over an interval; delivery is best effort and delayed by minutes |
RFC 3954 notes that NetFlow v9 was submitted as a basis for IETF work, and the IPFIX Working Group built IPFIX on it, so the two share concepts and many field numbers. Most modern collectors accept both.
How flow export works
The Cisco Flexible NetFlow model is a useful way to understand any exporter:
- Flow record: a set of key fields (with
matchstatements) that define what makes a new flow, and non-key fields (withcollectstatements) such as byte and packet counters. - Flow monitor: applied to an interface; holds a cache built from the record, ages out entries using active and inactive timeouts, and exports aged-out entries.
- Flow exporter: sends cache contents to one or more collectors; one exporter can serve several monitors and a monitor can use several exporters.
- Sampler: analyses only a subset of packets to reduce device load, at the cost of accuracy.
On the collector side, RFC 7011 requires a collecting process to keep template information for the duration of each transport session and not carry templates over between sessions. Over UDP there is no session, so templates are resent periodically and a collector may need to buffer data records that arrive before their template. Template withdrawal does not apply over UDP. These rules explain the common symptom of "missing data for the first few minutes" after a collector restart.
Fields that matter for security
IPFIX Information Elements are registered by IANA. The core set used by most detections:
| Information Element | IANA ID | Security use |
|---|---|---|
| sourceIPv4Address / destinationIPv4Address | 8 / 12 | Who talked to whom; matching against threat intelligence and asset inventory |
| sourceIPv6Address / destinationIPv6Address | 27 / 28 | The same for IPv6, often missing from older templates |
| sourceTransportPort / destinationTransportPort | 7 / 11 | Service identification; non-standard ports; scanning patterns |
| protocolIdentifier | 4 | TCP, UDP, ICMP and others; unusual protocols for tunnelling |
| octetDeltaCount / packetDeltaCount | 1 / 2 | Volume for exfiltration and denial of service detection |
| tcpControlBits | 6 | SYN-only flows for scans; RST patterns for rejected connections |
| ingressInterface / egressInterface | 10 / 14 | Direction and path through the device |
| ipClassOfService | 5 | Traffic class; occasionally abused for covert signalling |
| applicationId | 95 | Application classification where the exporter supports it |
| flowEndReason | 136 | Why the flow was exported (idle timeout, active timeout, end detected), which affects how long flows are stitched |
In AWS, the default VPC Flow Logs version 2 format is version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status and cannot be changed. A custom format adds fields such as pkt-srcaddr and pkt-dstaddr (the original addresses behind a NAT gateway or EKS pod), tcp-flags (FIN 1, SYN 2, RST 4, SYN-ACK 18, OR-ed across the interval), flow-direction, traffic-path, vpc-id, instance-id and reject-reason. The default maximum aggregation interval is 600 seconds with a 60 second option, and AWS states delivery typically takes about 5 minutes to CloudWatch Logs and about 10 minutes to S3. For cloud log design see the Cloud Logging and Audit Trail Guide.
Designing collection
- Where to export: internet edge (all ingress and egress), data centre and campus cores, links between security zones, VPN concentrators, and every cloud VPC or virtual network. Zone boundaries matter most because lateral movement crosses them. See the Network Segmentation Guide.
- Sampling: use unsampled flow where devices can sustain it, especially at edges and on server segments. If sampling is unavoidable on very high-speed links, record the rate so analytics can scale counts, and accept that low-volume flows will be missed.
- Timeouts: shorter active timeouts give fresher data for long-lived connections (useful for detecting persistent tunnels) at the cost of more records. Check your platform's defaults rather than assuming them.
- Bidirectional records: decide whether the collector stitches the two directions of a conversation; detections such as ratio of bytes out to bytes in depend on it.
- Enrichment: join flows with DHCP, asset inventory, identity (which user or workload owned an IP at that time), DNS logs and threat intelligence. Raw IP pairs are hard to act on.
- Retention: flows are compact compared with packets, so organisations commonly keep them far longer than packet captures; set a period that covers your typical dwell time and investigation needs. See the Log Management and Retention Guide.
| Option | Best for | Trade-off |
|---|---|---|
| Flow only | Broad coverage, long retention, cloud and branch sites | No payload; limited application detail |
| Flow plus targeted packet capture | Most enterprises: flow everywhere, packets at the edge and critical segments | Two pipelines to run |
| NDR sensor generating rich metadata | Deep protocol analysis (for example TLS and DNS details) on key links | Sensor cost and placement limits; see the NDR Guide |
Small organisations: enable flow export on the internet firewall and cloud flow logs, send both to the SIEM, and start with a handful of detections. Large organisations: run a dedicated collector tier close to exporters, normalise NetFlow, IPFIX, sFlow and cloud logs to one schema, and feed both the SIEM and an analytics store sized for months of data.
Implementation: a phased rollout
- Inventory exporters. List devices and cloud accounts that can export flow, the formats they support (v5, v9, IPFIX, sFlow) and whether they support IPv6 fields.
- Build the collector path. Place collectors on a management network, restrict inbound export traffic by source address with ACLs, and size storage from a sample of exported volume.
- Pilot one boundary. Enable export on the internet edge with a full template (addresses, ports, protocol, counts, TCP flags, interfaces, timestamps) and validate against firewall logs for the same period.
- Expand by zone. Add data centre cores, zone boundaries and cloud VPCs. In AWS, use a custom format with
pkt-srcaddr,pkt-dstaddrandtcp-flagsand the 60 second interval where cost allows. - Add enrichment and detections. Join with asset, identity and DNS data, then deploy detections in alert-only mode and tune before paging anyone.
- Test continuity. Restart a collector and confirm templates are reacquired; simulate an exporter failure and confirm the "no data" alert fires. Rollback for a device is removing the monitor from the interface, which stops export without affecting forwarding.
Securing the export path
Flow data reveals the whole shape of the network, and the export protocol is usually unauthenticated UDP, so an attacker on the path can read it and anyone who can reach the collector can inject false records or templates. Controls:
- Isolate transport: export over an out-of-band management network or a dedicated VRF, never across the internet in clear text.
- Restrict sources: collectors accept export only from known exporter addresses, enforced by host firewall and network ACLs.
- Use secure transport where available: RFC 7011 covers TLS and DTLS with mutual authentication; IANA assigns port 4739 for IPFIX and 4740 for IPFIX over TLS or DTLS. Support varies widely by device, so check before designing around it.
- Protect the collector: harden it like any log platform, with role-based access, because flow history is valuable reconnaissance for an intruder.
- Watch for gaps: an exporter that goes silent can be a failure or a deliberate attempt to blind monitoring.
Where identity fits
Flows answer "which IP", and investigations need "which identity". Map IPs to users through VPN and DHCP logs, and to workloads through cloud instance and pod metadata (the AWS instance-id and pkt-srcaddr fields help). Service accounts and workload identities generate most east-west traffic, so baselining flows per workload makes misuse of a stolen machine credential visible: a build agent suddenly talking to a database it never used is a strong signal. Collector and exporter management accounts, and any API keys used to pull cloud flow logs, are themselves privileged non-human identities. See the Cloud Logging and Audit Trail Guide.
Detections and measurement
| Behaviour | Flow pattern | ATT&CK |
|---|---|---|
| Internal scanning | One source, many destinations or ports, SYN-only or rejected flows | T1046 Network Service Discovery |
| Beaconing | Regular, small flows from one host to the same external destination; see the Beaconing and C2 Detection Guide | T1071 Application Layer Protocol; T1571 Non-Standard Port |
| Bulk exfiltration | Unusual outbound byte volume or out/in ratio to a new destination | T1041 Exfiltration Over C2 Channel; T1048 Exfiltration Over Alternative Protocol |
| Tunnelling over unusual protocols | Large or persistent ICMP or other non-TCP/UDP flows | T1095 Non-Application Layer Protocol |
| Proxy and relay use | Internal host relaying traffic between other hosts and the internet | T1090 Proxy |
| Denial of service | Sudden packet or flow rate spikes to one destination | T1498 Network Denial of Service |
Metrics:
- Percentage of zone boundaries and cloud VPCs exporting flow (target 100 per cent of in-scope).
- Exporter health: number of exporters silent for more than the expected interval.
- Template decode errors per day at the collector.
- Percentage of flows enriched with an asset or identity owner.
- Median time from flow export to availability in the SIEM.
Using flow data in investigations
During an incident, flow history is often the only record of what a compromised host talked to before the endpoint agent was installed or after it was disabled. Typical questions it answers quickly:
- Scope: which internal hosts did the compromised system connect to, on which ports, and when did that start?
- First contact: when did any internal host first talk to a known malicious address or a newly seen destination?
- Volume: roughly how much data left through a given destination, which feeds breach notification decisions. Flow volume shows bytes moved, not what they contained.
- Containment check: after blocking, did traffic to the attacker's infrastructure actually stop, or move to a new address?
Keep timestamps in UTC across exporters and collectors, and record exporter clock sources, because flow timelines are joined with endpoint and identity logs. The Data Exfiltration Playbook shows where flow evidence fits in a response.
Common failure patterns
- Edge only: exporting at the internet firewall but not internally, so lateral movement between segments is invisible.
- Silent heavy sampling: sampling at high ratios without recording the rate, then treating counts as exact.
- IPv4-only templates: IPv6 traffic not exported at all, leaving a blind protocol.
- Unprotected export: UDP export across untrusted networks, or collectors open to any source.
- No enrichment: analysts staring at IP pairs with no owner, so alerts are never closed.
- Ignoring cloud: on-premises flow is collected but VPC flow logs are off or use the default format without NAT-translated addresses.
Framework mapping
| Framework | Reference | How it applies |
|---|---|---|
| NIST SP 800-53 Rev 5 | SI-4 System Monitoring; AU-12 Audit Record Generation; SC-7 Boundary Protection | Generating and analysing flow records at boundaries |
| NIST CSF 2.0 | DE.CM-01; DE.AE-02 | Monitoring networks and analysing adverse events |
| CIS Controls v8.1 | Control 13 Network Monitoring and Defense | Collecting network traffic flow logs; see the CIS Control 13 Network Monitoring and Defense Guide |
| MITRE D3FEND | D3-NTA Network Traffic Analysis; D3-NTCD Network Traffic Community Deviation; D3-PMAD Protocol Metadata Anomaly Detection | Defensive techniques built on flow data |
Practitioner checklist
- Inventory exporters and the flow formats and IPv6 support of each.
- Export unsampled flow at the internet edge, zone boundaries and data centre cores where possible.
- Enable cloud flow logs in every VPC with a custom format including original addresses and TCP flags.
- Place collectors on a management network and restrict export sources.
- Use TLS or DTLS for IPFIX where devices support it.
- Enrich flows with asset, identity, DHCP and DNS data.
- Deploy scanning, beaconing, exfiltration and tunnelling detections in alert-only mode, then tune.
- Alert on silent exporters and template decode errors.
- Set retention to cover your investigation window and test retrieval.
Standards and references
- IETF: RFC 7011 Specification of the IPFIX Protocol for the Exchange of Flow Information
- IETF: RFC 3954 Cisco Systems NetFlow Services Export Version 9
- IANA: IP Flow Information Export (IPFIX) Entities registry
- AWS: Flow log records
- Cisco: Getting Started with Flexible NetFlow
- sFlow.org: About sFlow
- MITRE D3FEND: Network Traffic Analysis
Related NHI Mgmt Group resources: Network Monitoring and Packet Capture Guide · NDR Guide · Beaconing and C2 Detection Guide · Data Exfiltration Guide