Signature-Based vs Behavioural Detection: Where Each Wins and Fails

Signature-based detection catches what defenders already know, behavioural detection catches what looks wrong, and attackers deliberately work in the gap between the two. Commodity malware is still stopped at scale by signatures, while intrusions built on stolen credentials, built-in administration tools and over-privileged service accounts leave no known-bad artefact to match. This guide compares the two approaches by the threats each catches, how each fails, what data and tuning each needs, and how to layer them by asset type. It is written for detection engineers, SOC leads and security architects deciding where to invest. It goes beyond product categories, which the EDR vs Antivirus guide covers, and beyond analytics design, which the UEBA Guide covers.

Key takeaways

• Signatures are precise but backward-looking. NIST SP 800-94 describes signature-based detection as very effective against known threats and largely ineffective against unknown threats, evasion and many variants.
• Behavioural detection finds the unknown at the cost of noise. The same NIST guide names a high false positive rate and difficulty explaining alerts as the main weaknesses of anomaly-based detection.
• Credential misuse is behavioural territory. A valid login with stolen credentials matches no signature, so baselines of identity, location, timing and access patterns are the main signal.
• Baselines can be poisoned. Activity present during training becomes "normal", and dynamic profiles can be shifted by an attacker who increases activity slowly.
• Layer by asset, not by fashion. File-heavy endpoints and mail gateways lean on signatures; identity providers, cloud control planes and service accounts lean on behaviour; most assets need both.

The short answer

Use signatures wherever a known-bad artefact exists and precision matters: malware files, exploit traffic, known malicious domains and command lines. Use behavioural detection wherever the attacker can look legitimate: valid accounts, administration tools, service accounts and API keys used out of pattern. Neither replaces the other. A programme with only signatures misses living-off-the-land and credential attacks; a programme with only behavioural analytics drowns in alerts that a cheap signature would have closed with certainty.

What each approach is

Signature-based detection compares observed events, such as packets, files or log entries, against patterns of known threats. NIST SP 800-94 calls it the simplest method, based on comparisons, with little understanding of protocol state, so it usually misses attacks made of several events that each look harmless. Signatures range from exact file hashes at the brittle end to rules on structure, behaviour sequences and command line patterns at the durable end; the Pyramid of Pain Guide explains why the durable end costs attackers more to evade.

Behavioural detection compares current activity with a profile of normal behaviour for a user, host, connection, application or workload and flags significant deviations. NIST SP 800-94 notes that profiles are built over a training period, typically days and sometimes weeks, and may be static (fixed until regenerated) or dynamic (adjusted continuously). MITRE D3FEND's User Behavior Analysis (D3-UBA) technique describes the same idea for people, with child techniques such as User Geolocation Logon Pattern Analysis (D3-UGLPA), Resource Access Pattern Analysis (D3-RAPA) and Authentication Event Thresholding (D3-ANET).

Two hybrid forms sit between the poles. Behavioural signatures are rules on known-bad behaviour (an office application spawning a script interpreter, a process reading LSASS memory) rather than known-bad bytes; they are precise but still only catch what someone thought to write. Machine learning classifiers trained on labelled malicious and benign samples behave more like generalised signatures than baselines: they generalise from known examples rather than from your environment's normal.

Side-by-side comparison

Threat models: what each catches

Known commodity threats

Ransomware loaders, banking trojans, infostealers and widespread exploit kits are produced at volume and reused. Signatures in antivirus engines, YARA rules and network IDS rules catch them cheaply and with low false positive rates, and can block before execution. Behavioural detection adds little here except as a backstop when a signature is late.

Living off the land

Joint guidance from CISA, NSA, FBI, ASD's ACSC, NCSC-UK and other agencies, published on 8 February 2024, explains that actors using built-in tools blend into normal administration and can change file names, paths, command line syntax and hashes to bypass known-bad detections. The same guidance recommends detailed and centralised logging, baselines of tools, account behaviour and network traffic, and behavioural analytics to separate malicious use from routine IT work. Behavioural signatures on process lineage help, but a baseline of who normally runs which tool on which host is what separates the administrator from the intruder. See the Living off the Land Guide.

Credential misuse

Valid Accounts (ATT&CK T1078), and Cloud Accounts (T1078.004) in particular, generate events that are individually legitimate: a successful sign-in, a query, a download. The Snowflake breach shows the pattern: infostealer credentials were used against accounts without MFA, there was no vulnerability in the platform, and so there was no exploit or file to match. Detection depended on noticing unusual source addresses, clients and query volumes. See the MITRE ATT&CK T1078 Valid Accounts Guide.

Non-human identity anomalies

Service accounts, API keys and workload identities are where behavioural detection works best, because their behaviour is far more regular than a human's: the same source, the same calls, the same schedule. A key used from a new network, calling an API it has never called, or reading far more data than usual is a strong signal. Signatures help only at the edges, for example a rule for a leaked key format in a public repository. See the Service Account Behavioural Baselining Guide and the Detection Engineering for Non-Human Identities Guide.

Failure modes

Data and tuning requirements

• Signature data: the objects themselves (files, packets, events) at the point of inspection, plus a reliable feed of rule updates. Coverage depends on inspection points: a signature engine that never sees encrypted traffic, or a file that never touches a scanned path, cannot match.
• Signature tuning: narrow rules on known false positive sources, staged rollout in alert mode, and retirement of rules that no longer earn their cost. Precision should be high enough to automate response.
• Behavioural data: consistent, long-retention telemetry per entity: authentication, process creation with command lines, cloud audit logs, network flows, data access. The joint living-off-the-land guidance stresses that default logging often misses key events, so enhanced logging is a prerequisite, not an extra.
• Behavioural context: identity attributes (role, department, manager), asset criticality and peer groups. D3FEND's Job Function Access Pattern Analysis (D3-JFAPA) technique compares access with profiles grouped by job function, which only works if HR and identity data are joined to the logs.
• Behavioural tuning: thresholds per entity type, suppression for known schedules, feedback from analyst verdicts, and a re-training plan. Expect weeks of tuning before alerts are trusted.

Layering decision by asset type

Small organisations should take signatures from managed endpoint and email products and spend scarce analyst time on a few high-value behavioural detections: identity provider sign-in anomalies and service account misuse. Large organisations can run full UEBA and their own signature content, and should invest in correlation that combines weak behavioural signals with signature hits on the same entity. For how identity threat detection and UEBA divide this work, see ITDR vs UEBA.

Worked scenario

An attacker buys a contractor's credentials from an infostealer log, signs in to the VPN and runs built-in tools to enumerate the domain, then uses a service account password found in a script to reach a database server. No file is dropped. Signature controls see nothing until the attacker downloads a known credential dumping tool, which the endpoint agent quarantines. Behavioural controls flag three weak signals earlier: the contractor signing in from a new country at an unusual hour, the contractor's account running enumeration commands it has never run, and the service account authenticating interactively from a workstation for the first time. Individually each might be dismissed; correlated per entity, they justify an investigation hours before the signature hit. The lesson is not that signatures failed but that each layer saw a different stage.

How to implement a layered approach

1. Inventory coverage: map current signature and behavioural detections to ATT&CK techniques and to the asset types above. Mark techniques covered by only one approach.
2. Fix logging first: enable command line, script and cloud audit logging and centralise it, because behavioural detection cannot start without history.
3. Harden signatures: move from hashes to structural and behavioural rules, retire stale content and add positive tests.
4. Start behavioural on regular entities: service accounts and API keys first, then privileged users, then the general user base, because regular behaviour gives the best signal to noise ratio.
5. Run in alert-only mode: let baselines train on a reviewed period and tune for several weeks before alerts page anyone.
6. Correlate: combine behavioural scores and signature hits per entity so that weak signals reinforce each other.
7. Roll back safely: keep previous rule releases and baseline snapshots so that a bad update or poisoned baseline can be reverted.

Detection and measurement

• Precision per detection: true positives over total alerts, tracked separately for signature and behavioural content.
• Mean time to detect by approach: for confirmed incidents, which layer fired first and how long after initial access.
• Dual-layer technique coverage: share of priority ATT&CK techniques covered by both a signature and a behavioural detection.
• Baseline coverage: share of service accounts and privileged users with a trained, current baseline.
• Analyst load: behavioural alerts per analyst per shift, and the share closed as benign without action.

Common failure patterns

• Buying behavioural analytics before fixing logging: models trained on incomplete data produce confident nonsense.
• Training on a compromised period: an environment with an undetected intruder teaches the model that the intruder is normal.
• Excluding service accounts because they are noisy: the noise usually means the baseline is too coarse, and the exclusion hands attackers the accounts with the most access.
• Treating hash lists as a detection programme: hashes change with every rebuild and catch only exact copies.
• Paging on every anomaly: without per-entity risk aggregation, analysts learn to ignore behavioural alerts within weeks.

Framework mapping

Practitioner checklist

• Map every priority ATT&CK technique to signature and behavioural coverage and close single-layer gaps first.
• Enable and centralise command line, script, authentication and cloud audit logging before deploying behavioural analytics.
• Replace hash-only signatures with structural and behavioural rules, each with a positive test.
• Baseline service accounts, API keys and workload identities before the general user population.
• Train baselines only on reviewed periods and limit how fast dynamic profiles can drift.
• Never exclude a whole class of accounts from behavioural detection; tune per entity instead.
• Aggregate weak behavioural signals into per-entity risk scores before alerting.
• Track precision and time to detect separately for each approach and report dual-layer coverage.

Standards and references

• NIST: SP 800-94 Guide to Intrusion Detection and Prevention Systems
• ASD's ACSC, CISA and partners: Identifying and Mitigating Living Off the Land Techniques
• MITRE D3FEND: User Behavior Analysis
• MITRE ATT&CK: T1078 Valid Accounts
• MITRE ATT&CK: T1027.002 Software Packing
• MITRE ATT&CK: T1218 System Binary Proxy Execution

Related NHI Mgmt Group resources: UEBA Guide · EDR vs Antivirus · Detection Engineering Guide · Sigma vs YARA vs Snort Rules