IDS and IPS Guide: Detection, Placement, Tuning

Most intrusion detection and prevention deployments fail quietly: the sensor is running, the rules are loaded, and nobody trusts the alerts enough to act on them or to switch blocking on. An IDS watches traffic and raises alerts; an IPS sits in the traffic path and can drop or reset what matches. Both depend on the same three things: where the sensor sees traffic, which detection method and rules it applies, and how well the output is tuned for the network it protects. This guide covers detection methods, sensor types and placement, inline modes and failure behaviour, rule management in Suricata and Snort 3, tuning, the limits that TLS 1.3 and Encrypted Client Hello put on inspection, and the identity questions sensors raise. It is written for security engineers and SOC leads who run or are about to run IDS and IPS engines. Flow records, packet capture and retention are covered separately in the Network Monitoring and Packet Capture Guide.

Key takeaways

Placement decides coverage more than rule count: a sensor that never sees east-west or management traffic cannot alert on it, however good its ruleset.
Use more than one detection method because signatures miss unknown attacks, anomaly models are noisy, and stateful protocol analysis cannot flag abuse that stays within the protocol rules (NIST SP 800-94).
Decide fail-open or fail-closed on purpose before going inline: Suricata defaults most exception policies to drop-flow in IPS mode, while NFQUEUE bypass lets traffic through when the engine is down.
Tune with thresholds, suppression and rule selection, not by turning categories off wholesale, and promote rules to blocking only after they have run quietly in alert mode.
Treat sensors as privileged infrastructure: they see cleartext credentials, hold management and update credentials, and belong on a separate management network.

IDS, IPS and the four sensor types

NIST SP 800-94, the Guide to Intrusion Detection and Prevention Systems, remains the standing NIST reference: it was published in February 2007, and the Revision 1 draft from 2012 was retired without being finalised. It groups IDPS technologies into four types, and the distinction is still the right starting point for deciding what to deploy.

Type	What it watches	Strengths	Blind spots
Network-based (NIDS/NIPS)	Packets on network segments, usually at perimeter, data centre and segment boundaries	Sees many hosts from one sensor, protocol decoding, inline blocking	Encrypted payloads, traffic that never crosses the sensor, host-internal activity
Wireless	802.11 traffic and radio environment	Rogue access points, misconfigured clients, wireless protocol attacks	Application-layer content once traffic reaches the wired network
Network behaviour analysis (NBA)	Flow statistics and traffic patterns	Scans, unusual volumes, new communication pairs, policy violations	Payload-level exploits; usually detection rather than prevention
Host-based (HIDS/HIPS)	One host: logs, processes, file integrity, local network stack	Sees decrypted traffic and local effects of an attack	Only covers hosts where the agent runs; can be tampered with by an attacker with admin rights

In current product language, NBA has largely become network detection and response, and HIDS/HIPS has largely moved into endpoint agents. See the NDR Guide for behavioural analytics and the Endpoint Security Guide for the host side. For radio coverage see the Wi-Fi Security Guide. The rest of this guide concentrates on network IDS and IPS engines.

Detection methods

SP 800-94 describes three detection methodologies. Open-source engines combine them: Suricata and Snort 3 are signature engines with deep protocol parsers, and Zeek, which its own documentation says is not a classic signature-based IDS, is built for semantic, behavioural and anomaly analysis through its scripting language.

Method	How it works	Good at	Weak at
Signature-based	Compares traffic to known patterns (content matches, byte sequences, protocol fields)	Known exploits and malware command and control, low false positives when rules are precise	New attacks, variants and evasion; attacks made of several harmless-looking events
Anomaly-based	Builds a profile of normal behaviour during training, flags deviations	Previously unseen activity, new hosts or services, volume changes	Many false positives, hard-to-explain alerts, slow attacks absorbed into the baseline
Stateful protocol analysis	Compares each protocol state against a model of correct behaviour (command order, argument lengths)	Protocol abuse other methods miss	Resource-hungry, imperfect models for proprietary protocols, cannot flag valid requests used maliciously (for example a flood of legitimate calls)

Behavioural scripting: Zeek raises detections through its Notice framework. A script calls NOTICE with a notice type such as SSH::Password_Guessing, and the Notice::policy hook decides whether to log it (Notice::ACTION_LOG), raise an alarm (ACTION_ALARM) or email it (ACTION_EMAIL). Duplicate notices with the same identifier are suppressed for an interval, which is Zeek's built-in answer to alert floods.
Evasion: attackers change the format or timing of an attack (fragmentation, overlapping segments, unusual encodings) without changing its effect. SP 800-94 notes that engines counter this by processing traffic the same way the target does, which is why stream reassembly and protocol normalisation settings matter as much as rules.
Where machine learning fits: "smart" or AI-driven detection is a form of anomaly detection. It inherits the same trade-offs: training data must be clean, and analysts need an explanation they can act on.

Placement and inline modes

Passive sensors receive a copy of traffic; inline sensors forward it and can block. The choice changes failure behaviour, latency and who has to approve the deployment.

Deployment	How traffic reaches the engine	Can block?	Main risk
SPAN or mirror port	Switch copies traffic to a monitor port	No (alerts only, optional TCP resets)	Oversubscribed or dropped copies under load, missed errors and VLAN tags
Network tap	Hardware copies traffic at the physical layer	No	Cost and physical access; still passive
Cloud mirroring	Provider feature copies interface traffic to a collector	No	Per-interface setup and cost; check provider limits
Inline, layer 2 (Suricata AF_PACKET `copy-mode: ips`, Snort 3 `afpacket` DAQ with an interface pair and `-Q`)	Engine bridges two interfaces	Yes	Engine outage becomes a network outage unless bypass hardware is used
Inline, layer 3 (Suricata NFQUEUE)	Netfilter hands packets to Suricata through a queue, e.g. `iptables -I FORWARD -j NFQUEUE` with `suricata -q 0`	Yes	Traffic stops if the queue has no listener, unless `bypass` is set

Placement priorities for most organisations, in order:

Internet edge, inside the firewall: sees what passed the firewall rather than everything the internet throws at it, which keeps alert volume meaningful.
Remote access and edge appliances: traffic leaving VPN gateways and load balancers towards internal systems, because the appliances themselves are frequent targets and rarely run your agents.
Data centre and segment boundaries: server-to-server and user-to-server paths. Align sensors with your zones; see the Network Segmentation Guide.
Management networks: device administration, hypervisor and backup traffic, where attackers using stolen credentials look most like administrators.
Cloud VPCs and Kubernetes clusters: through traffic mirroring or a virtual inline appliance in a transit path.

Fail-open versus fail-closed

NFQUEUE bypass: with nftables queue num 3-5 options fanout,bypass, packets are accepted when no program is listening on the queue. The link stays up when Suricata stops, but nothing is blocked while it is down.
Suricata exception policies: when the engine hits an internal limit (stream or flow memcap, midstream sessions, application-layer parser errors), the exception-policy master switch and per-setting policies (memcap-policy, midstream-policy, error-policy) decide what happens. Values include drop-flow, drop-packet, pass-flow, pass-packet, bypass, reject and ignore. In IPS mode the default is drop-flow, so most policies fail closed; in IDS mode auto means the policies are ignored.
Decision rule: fail closed where an uninspected flow is worse than an outage (an internet-facing administration path), fail open where availability dominates (a busy customer-facing link). Write the decision down and test it by stopping the engine during a change window.

Rules and rule management

A Suricata rule has three parts: the action, the header (protocol, addresses, ports, direction) and options in parentheses separated by semicolons. Snort 3 uses the same header and body shape and ignores extra whitespace, so rules can be laid out across lines without backslashes.

Engine	Actions	Notes
Suricata	`alert`, `pass`, `drop`, `reject`, `rejectsrc`, `rejectdst`, `rejectboth`	`drop` only has effect in IPS mode; in IPS mode any reject action also drops
Snort 3	`alert`, `block`, `drop`, `log`, `pass`, `react`, `reject`, `rewrite`	`block` drops the packet and the rest of the flow; `drop` discards only the current packet; `rewrite` uses the `replace` option

A short illustrative local rule, written to flag HTTP Basic authentication crossing a segment where it should never appear (adjust variables and the SID range to your own local policy):

alert http $HOME_NET any -> $HOME_NET any (msg:"LOCAL Basic auth between internal services"; flow:established,to_server; http.header; content:"Authorization|3a 20|Basic "; nocase; threshold:type limit, track by_src, count 1, seconds 3600; classtype:policy-violation; sid:9000101; rev:1;)

Rule sources: suricata-update (bundled since Suricata 4.1) downloads Emerging Threats Open by default and merges rules into /var/lib/suricata/rules/suricata.rules. list-sources and enable-source add further sets. Snort 3 uses its own rule packages. Whatever the source, record which sets are enabled and why.
Local changes outside the rule files: keep enables and disables in enable.conf and disable.conf (by SID, group: file or re: pattern) so updates do not overwrite them. Keep local rules in their own file with a reserved SID range.
Change control: treat rule updates like code. Test new rulesets against stored captures before deploying, roll out to one sensor first, and keep the previous ruleset to roll back to.
Coverage reasoning: map enabled rules to ATT&CK techniques you care about, such as T1190 Exploit Public-Facing Application, T1071 Application Layer Protocol and T1046 Network Service Discovery. See the MITRE ATT&CK Coverage Mapping Guide.

Tuning: from noise to signal

SP 800-94 is blunt that accuracy cannot be perfect and that reducing false positives usually increases false negatives. Tuning is a standing task, not a project phase. Suricata gives several levers inside rules:

Keyword	Behaviour	Example	Use it for
`threshold: type limit`	At most count alerts per period	`threshold: type limit, track by_src, seconds 180, count 1;`	Noisy but useful policy rules
`threshold: type threshold`	Alert on every Nth match in a window	`threshold: type threshold, track by_src, count 10, seconds 60;`	Volume-based detections
`threshold: type both`	Alert once the count is reached, then cap	`threshold: type both, track by_src, count 5, seconds 360;`	Brute-force style patterns
`threshold: type backoff`	Alert at growing intervals per flow	`threshold:type backoff, track by_flow, count 1, multiplier 10;`	Long flows that keep matching
`detection_filter`	Alert on every match after the count is reached, without resetting	`detection_filter: track by_dst, count 10, seconds 60, unique_on dst_port;`	Scan detection by distinct ports

Thresholds and drops: for limit, backoff and both, drop and reject actions still apply to every matching packet, so you can cap alert volume without weakening a blocking rule.
Suppress narrowly: suppress a rule for one known scanner or monitoring host, never for a whole subnet, and give every suppression an owner and a review date.
Set address variables correctly: wrong $HOME_NET and $EXTERNAL_NET values are the most common reason rules either never fire or fire on everything.
Disable rules for absent technology: rules for platforms you do not run cost CPU and create noise; disabling them through disable.conf is legitimate tuning.
Promote to blocking deliberately: SP 800-94 describes simulation modes that report what prevention would have done. The same approach works with any engine: run a rule as alert, review a few weeks of hits, then convert to drop through a modification policy rather than editing vendor files.

Encryption: what an IDS can still see

TLS 1.3 (RFC 8446) encrypts all handshake messages after the ServerHello, including the server certificate, and removes static RSA key exchange, so every session has forward secrecy and passive decryption with a server private key no longer works. Encrypted Client Hello, published as RFC 9849 in March 2026, goes further: the real server name and ALPN list travel in an encrypted inner ClientHello, and passive observers see only an outer public name for the provider.

Still visible: addresses, ports, timing, sizes, the outer ClientHello and, where ECH is not used, the SNI. Suricata logs tls.sni and can compute JA3 and JA4 client fingerprints when app-layer.protocols.tls.ja3-fingerprints and ja4-fingerprints are enabled.
Lost: payload signatures, HTTP URIs and headers, file extraction, and (with ECH) reliable destination naming for content-delivery and hosting providers.
Options: decrypt at a proxy you control and feed the decrypted stream to the engine, move inspection to the endpoint, or rely on metadata and behavioural detection. Each has privacy and operational costs; see the TLS Inspection Guide.

Where identity fits

Cleartext credentials: network sensors still find passwords and tokens in FTP, Telnet, HTTP Basic, LDAP simple binds and SNMPv1/v2c community strings. Treat every such alert as a credential exposure: rotate the secret, then fix the protocol. See the Network Ports and Protocols Security Guide.
Service-to-service baselines: machine identities talk to a small, stable set of peers on fixed ports. An application server suddenly opening SSH (T1021.004) or a service account authenticating from a new subnet is a stronger signal than most signatures. See the Service Account Security Guide.
Sensor management credentials: sensors hold rule-update credentials, API keys for the SIEM or management console and sometimes TLS private keys for decryption. Keep them in a secrets manager, scope them per sensor and rotate them.
Management plane: SP 800-94 recommends a separate management network, with a management interface that cannot pass traffic to the monitoring interfaces; a VLAN is cheaper but offers less protection. Restrict console access with MFA and log every rule and policy change.

How to implement

Define the questions first: list the attacks you need to see (exploit attempts on edge services, lateral movement, command and control, cleartext credentials) and the segments where they would appear.
Place passive sensors: start with taps or mirror ports at the internet edge and the main data centre boundary. Confirm the sensor sees both directions and the expected volume, and check packet drop counters.
Baseline with a broad ruleset in alert mode: run for two to four weeks, set $HOME_NET, disable rules for absent technology, and add thresholds for chatty rules.
Integrate: send EVE JSON or unified output to the SIEM with asset and owner context. See the SIEM Guide.
Pilot inline on one link: choose fail-open or fail-closed, set exception policies explicitly, and keep all rules on alert at first.
Promote a small blocking set: high-confidence exploit and malware rules first, each with a named owner. Watch alert.action: blocked events daily for the first weeks.
Extend coverage: internal segments, management networks and cloud, then local rules for your own service-to-service policy.
Rollback: keep the previous ruleset and configuration under version control, document how to switch an inline sensor to bypass or tap mode, and rehearse it.

Detection and measurement

Monitor the sensors as well as their alerts. Suricata's EVE output records alert.action as allowed or blocked and a separate verdict for the final packet decision, which lets you report alerting and blocking separately.

Sensor health: packet drop rate per sensor and interface, kept below an agreed level.
Coverage: percentage of defined zones and internet egress points with a working sensor.
Signal quality: percentage of alerts closed as true positive or useful, tracked per rule category.
Tuning debt: number of suppressions past their review date.
Response: median time from high-severity alert to triage.
Blocking safety: blocked events later confirmed as legitimate traffic, per month.

Alert on the sensors going quiet: a sensor that stops sending events, a sudden drop in decoded protocols or a rule-update failure is often the first sign of a broken span or a tampered engine (ATT&CK T1685 Disable or Modify Tools).

Framework mapping

Framework	Reference	Relevance
NIST CSF 2.0	DE.CM-01, DE.AE-02, DE.AE-03	Networks monitored for adverse events; events analysed and correlated
NIST SP 800-53 Rev. 5	SI-4, SI-4(2), SI-4(4), SC-7	System monitoring, automated real-time analysis, inbound and outbound traffic monitoring, boundary protection
CIS Controls v8.1	13.2, 13.3, 13.7, 13.8, 13.11	Host-based and network intrusion detection and prevention, tuning alert thresholds
ISO/IEC 27001:2022 Annex A	A.8.16, A.8.20	Monitoring activities; networks security
MITRE D3FEND	D3-NTSA, D3-NTA, D3-ITF	Network traffic signature analysis, network traffic analysis, inbound traffic filtering
MITRE ATT&CK	T1190, T1046, T1071, T1040, T1685	Exploitation, discovery, command and control, sniffing, and attempts to disable sensors

See the MITRE D3FEND Guide for how defensive techniques are modelled.

Common failure patterns

Trusting the appliance to report on itself: in the Ivanti Connect Secure exploitation, CISA warned that the vendor's integrity checker failed to detect compromise. Sensors that watch the traffic leaving edge appliances provide an independent view the device cannot falsify.
No sensor on the management plane: in the Salt Typhoon telecom intrusions, Cisco Talos described configurations pulled over TFTP and FTP, SNMP and TACACS+ traffic captured for credentials, and SSH enabled on high ports. Each is detectable on the wire, but only where a sensor sees device management traffic.
Inline without a failure plan: an engine that runs out of memory or crashes either drops a production link or silently stops blocking, and the team finds out which during an incident.
Tuning by deletion: whole rule categories disabled to stop noise, leaving gaps nobody recorded.
Stale address variables: new cloud ranges and subnets never added to $HOME_NET, so internal traffic is treated as external or ignored.
Rule updates failing silently: expired update credentials or proxy changes leave sensors on months-old rules.

Practitioner checklist

Document which segments, links and cloud networks each sensor sees, and check both traffic directions arrive.
Choose and record fail-open or fail-closed per inline link; set Suricata exception policies explicitly.
Run new rules in alert mode first and promote to blocking with a named owner.
Keep enable, disable and local rules in separate files under version control.
Give every suppression an owner and a review date.
Write local rules for cleartext credentials and unexpected service-to-service protocols.
Put sensor management on a separate network, behind MFA, with logged changes.
Store sensor update and integration credentials in a secrets manager and rotate them.
Alert on sensor silence, packet drops and failed rule updates.
Review encryption coverage yearly as TLS 1.3 and ECH adoption grows.

IDS and IPS Guide: Detection Methods, Placement and Tuning