Next-Generation Antivirus (NGAV) Guide: How It Works and Where It Fits

Signature-based antivirus finds malware that someone has already seen, and most of the malware that hurts organisations today is new, packed, customised or not a file at all. Next-generation antivirus (NGAV) adds machine learning, behaviour monitoring, cloud lookups and exploit and script controls to the prevention layer on the endpoint. This guide explains how NGAV actually decides what to block, how it differs from legacy antivirus and from EDR, how to deploy and tune it without breaking business applications, and how attackers try to switch it off. It is written for endpoint engineers, SOC leads and architects replacing or hardening an existing antivirus estate. Detection and response telemetry is covered in the EDR Guide, and the wider endpoint stack in the Endpoint Security Guide.

Key takeaways

• NGAV is a prevention layer, not a detection platform. It decides block or allow on the endpoint in milliseconds; investigation, hunting and response need EDR or XDR alongside it.
• Signatures still matter, but they are one engine of several. Static machine learning, behaviour monitoring, cloud reputation and attack surface rules catch what signatures miss, and NIST SP 800-83 already noted that customised malware may never reach a signature database.
• Exclusions are the most common self-inflicted gap. Every folder, process or extension you exclude is a place malware can run unscanned, so exclusions need owners, expiry dates and review.
• Attackers target the agent first. Ransomware operators routinely disable antivirus before deploying encryptors, so tamper protection and alerting on protection state changes are as important as detection quality.
• Test in audit mode, measure, then enforce. Behaviour and attack surface rules break legitimate tools if switched straight to block; pilot rings and a documented rollback path keep the rollout safe.

From legacy antivirus to NGAV: what changed

Legacy antivirus compares files against a database of known bad patterns (hashes and byte signatures) during on-access and scheduled scans. NIST SP 800-83 Rev. 1 describes the classic capabilities: scanning startup files and boot records, on-access scanning of each file as it is downloaded, opened or executed, monitoring email clients and browsers, and disinfecting or quarantining infected files. The same publication is explicit about the limit: antivirus that looks for characteristics of known malware is less effective against unknown threats, new threats can take hours or days to reach the database, and highly customised malware may never appear in it.

NGAV keeps signatures for cheap, high-confidence blocking of commodity malware, and adds engines that judge a file or action on its properties and behaviour rather than its identity.

Most current products ship NGAV and EDR in one agent, often under the label endpoint protection platform (EPP) for the prevention part. The distinction still matters when you design policies and staffing: prevention settings are owned by endpoint engineering, detection and response by the SOC. See the EDR Guide and the XDR Guide.

How NGAV decides: the detection engines

• Signatures and hash reputation: exact or fuzzy matches against known malware. Fast and low in false positives, blind to new samples. D3FEND models this as File Hashing (D3-FH) and File Content Rules (D3-FCR).
• Static machine learning: a model scores a file before execution from features such as header structure, imports, entropy, strings and packer traits. It catches repacked variants of known families but produces false positives on unusual in-house tools, installers and packed legitimate software.
• Emulation and sandboxing: the engine runs suspicious code in an emulator or a cloud detonation environment to unpack it and observe behaviour. D3FEND lists Emulated File Analysis (D3-EFA) and Dynamic Analysis (D3-DA).
• Behaviour monitoring: the agent watches what a running process does: spawning shells from Office, injecting into other processes, mass file renames, deleting shadow copies, reading LSASS memory. This is D3FEND Process Spawn Analysis (D3-PSA) and Process Analysis (D3-PA), and the response is often Process Termination (D3-PT).
• Cloud lookups: the agent sends metadata, and with consent sample files, to the vendor cloud for a verdict based on global prevalence and larger models. This is what lets an NGAV block a file the first time it is seen anywhere, at the cost of a dependency on connectivity and on data leaving the endpoint.
• Script and memory scanning: inspection of PowerShell, VBScript, JavaScript and macro content after de-obfuscation (on Windows through the Antimalware Scan Interface, AMSI), which targets T1059.001 PowerShell and T1027 Obfuscated Files or Information.
• Attack surface and exploit controls: rule sets that block classes of behaviour regardless of the file, such as Office applications creating child processes or credential theft from LSASS.
• Ransomware-specific controls: controlled folder access, canary files and rollback features that watch for T1486 Data Encrypted for Impact and T1490 Inhibit System Recovery.

Worked example: Microsoft Defender Antivirus settings

Microsoft Defender Antivirus is built into Windows and is a useful reference because its controls are publicly documented. The same concepts exist under other names in other products.

Attack surface reduction rules run in Audit, Block or Warn mode, and the advanced hunting action types (for example AsrLsassCredentialTheftAudited and AsrLsassCredentialTheftBlocked) let you measure the effect of a rule before enforcing it. See the Microsoft Defender for Endpoint Guide for the full platform.

Antivirus exclusions: the gap you create yourself

Exclusions exist because scanning can slow databases, build servers and backup jobs, and because NGAV models sometimes misjudge in-house software. Attackers know this: an excluded folder such as a temp or tools directory is an ideal staging location, and some intrusions add their own exclusions once they hold admin rights. Tuning is covered in depth in the EDR Tuning and Exclusions Management Guide; the rules below are the minimum.

• Owner and expiry: every exclusion has a named owner, a ticket and a review date. Remove exclusions for software that has been retired.
• Central only: manage exclusions from the console or MDM, not locally; with Microsoft tamper protection on, local exclusions cannot be added at all.
• Visibility: report exclusions per device group monthly and alert on any exclusion created outside the change process.

Tamper protection and antivirus evasion

Disabling the security agent is a standard step in hands-on intrusions. CISA's advisory on Blacksuit (Royal) ransomware describes actors disabling antivirus before exfiltration and encryption, in one case by modifying Group Policy Objects from a domain controller with a legitimate admin account, and using tools that kill processes or uninstall security software. ATT&CK v20 tracks this under the Defense Impairment tactic (TA0112) as T1685 Disable or Modify Tools.

• Agent disable or uninstall: by an admin account, a GPO, a removal tool or a stop command. Defence: tamper protection, uninstall passwords or tokens, and treating the agent console as a privileged system.
• Vulnerable driver abuse (bring your own vulnerable driver): a signed but vulnerable driver is loaded to terminate protected processes from kernel mode, using T1068 Exploitation for Privilege Escalation. Defence: driver blocklists, the vulnerable signed driver ASR rule, hypervisor-protected code integrity.
• Exclusion abuse: adding exclusions or dropping payloads into existing ones. Defence: tamper-protected exclusions and alerts on configuration change.
• Living off the land: using signed system binaries (T1218 System Binary Proxy Execution) and scripts so there is no malicious file to scan. Defence: behaviour rules, script scanning, application control. See the Living off the Land Guide.
• Packing, obfuscation and process injection: T1027 and T1055 Process Injection hide payloads from static engines. Defence: emulation, memory scanning, behaviour monitoring.
• Safe Mode and offline tampering: rebooting into a mode where the agent does not run, or encrypting from an unmanaged virtual machine. Defence: restrict boot configuration changes and monitor for unexpected reboots into Safe Mode.

"Antivirus evasion tools" sold or shared by criminals are packaged versions of these techniques. Testing your own NGAV against them is a job for authorised red teams; the defensive answer is layered controls rather than hoping a single engine catches every crypter. The EDR Evasion and Defence Guide and Top 10 EDR Evasion Techniques go deeper.

Rogue antivirus is a separate problem: fake security software (scareware) that shows false infection warnings to push users into paying or installing malware. Managed fleets defeat it with application control and browser protections; it is a user awareness topic for unmanaged and home devices.

Where identity fits

Most real-world NGAV bypasses do not defeat the engine, they use an account that is allowed to change it. Local administrator rights let an attacker stop services or add exclusions, domain admin rights let them push a GPO that disables protection, and a compromised console administrator can change policy for the whole fleet. Remove standing local admin rights (see the Endpoint Privilege Management Guide), protect the NGAV console with phishing-resistant MFA and role-based access, and treat console API keys and deployment service accounts as privileged non-human identities with rotation and monitoring.

Deploying or replacing antivirus: a phased approach

An antivirus replacement project is mostly change management. NIST SP 800-83 recommends starting application whitelisting in audit mode because enforcement blocks legitimate software that is missing from the list; the same logic applies to NGAV behaviour and ASR rules.

1. Inventory and baseline: list every operating system and version, server roles, VDI pools, legacy systems and existing exclusions. Record current detection volumes and false positive rates so improvement can be shown.
2. Design policy tiers: separate policies for workstations, servers, developer machines, VDI and high-value assets. Decide cloud block level, sample submission (a privacy decision) and ASR rule set per tier.
3. Coexistence plan: two real-time engines on one host cause performance problems and conflicts. Put the new product in passive mode or remove the old one per ring, and confirm the old agent is really uninstalled, not just disabled.
4. Pilot ring in audit mode: deploy to IT and a representative business group for two to four weeks with behaviour and ASR rules in audit. Review every audit hit, decide block, exclude narrowly, or leave in audit.
5. Enable tamper protection early: as soon as the agent is stable in a ring, so that the transition window does not become an easy target.
6. Staged enforcement: move rules to block ring by ring (pilot, early adopters, broad, servers last). Keep a documented rollback per rule: switch it back to audit for the affected group, not off for everyone.
7. Connect to operations: forward detections and protection state events to the SIEM or XDR, define alert handling, and give the service desk a false positive process. See the SIEM Guide.
8. Validate: run benign test files (such as the EICAR test string) and authorised attack simulations mapped to ATT&CK, and confirm both the block and the alert arrive.

Choosing and evaluating NGAV

• Built-in versus third party: operating system antivirus (for example Defender on Windows) removes agent conflicts and is often licensed already; a third-party agent may suit mixed estates or a single cross-platform console. Choose on management model, platform coverage and SOC integration rather than on headline detection claims.
• Small organisations: a well-configured built-in engine with cloud protection, tamper protection and ASR rules, managed centrally, plus an MDR service watching alerts, usually beats an advanced product nobody tunes. See the MDR Guide.
• Large organisations: policy tiering, role-based console access, API access for automation, and server and Linux coverage matter more than marginal detection differences.
• Independent tests: read the methodology, not just the chart. AV-Comparatives' Real-World Protection Test uses live malicious URLs, judges compromise from independent monitoring of file, registry, process and network changes, rates cases where the user was asked as "user-dependent", and measures false positives against about 1,000 popular domains and about 100 clean applications. AV-TEST's protection module combines a real-world 0-day stage with a detection stage on recent reference samples. Both test default consumer or business configurations, which may not match yours.
• Offline behaviour: ask what protection remains when the device cannot reach the cloud, and how fast models update.

The EDR and XDR Buyer's Guide covers procurement questions.

Detection, alert response and metrics

An NGAV block is not the end of an incident. A detection means something reached the endpoint, and a failed remediation means it may still be running. On Windows, the Microsoft-Windows-Windows Defender/Operational log carries the key events.

Follow the NIST SP 800-83 incident phases for confirmed malware: containment (isolate the host, block the hash fleet-wide), eradication, recovery and lessons learned. The Incident Response Guide covers the process.

• Coverage: percentage of managed endpoints with the agent installed, healthy and in active (not passive) mode, target above 98 percent.
• Protection state: percentage of devices with tamper protection, cloud protection and real-time protection on.
• Definition and model freshness: devices with intelligence older than 3 days.
• Exclusion hygiene: number of exclusions, percentage with owner and expiry, exclusions on user-writable paths (target zero).
• Remediation failure rate: 1118 and 1119 events as a share of detections, and median time to isolate.
• False positive rate: confirmed false positives per 1,000 endpoints per month, by rule.

Framework mapping

Common failure patterns

• Bought NGAV, configured legacy AV: cloud protection, behaviour rules and ASR left off or in audit forever, so the product behaves like signature antivirus.
• Exclusion sprawl: years of broad folder and process exclusions added for performance tickets and never removed, including user-writable paths.
• No alert on protection disabled: events such as 5001 and 5007 are not forwarded, so an attacker turning protection off before encryption, as in the Blacksuit intrusions CISA describes, goes unnoticed.
• Admins can switch it off: tamper protection disabled for "troubleshooting" and never re-enabled, or GPO and console admins with standing access. Whoever holds those identities, including an intruder, holds the off switch.
• Blind spots: servers, Linux hosts, VDI golden images and legacy systems left with old agents or none at all.
• Treating a block as closure: quarantined files closed without asking how the malware arrived or whether a failed remediation left it running.

Practitioner checklist

• Confirm every managed endpoint, including servers and Linux, runs a healthy NGAV agent in active mode.
• Enable cloud protection, behaviour monitoring and script scanning on all tiers.
• Turn on tamper protection centrally and check it per device (for Defender, IsTamperProtected).
• Review every exclusion: owner, ticket, expiry, and no user-writable paths.
• Pilot attack surface and behaviour rules in audit mode, then enforce ring by ring with a per-rule rollback.
• Forward detection, remediation failure and protection state events to the SIEM or XDR and alert on protection disabled.
• Remove standing local admin rights and protect the NGAV console with phishing-resistant MFA and RBAC.
• Validate quarterly with test files and authorised ATT&CK-mapped simulations, and track the metrics above.

Standards and references

• NIST: SP 800-83 Rev. 1, Guide to Malware Incident Prevention and Handling for Desktops and Laptops
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls (SI-3 Malicious Code Protection)
• CISA: #StopRansomware: Blacksuit (Royal) Ransomware
• MITRE ATT&CK: Disable or Modify Tools (T1685)
• Microsoft Learn: Protect security settings with tamper protection
• Microsoft Learn: Attack surface reduction rules reference
• Microsoft Learn: Microsoft Defender Antivirus event IDs and error codes
• AV-Comparatives: Real-World Protection Test methodology

Related NHI Mgmt Group resources: Endpoint Security Guide · EDR Guide · Windows Endpoint Hardening Guide · Ransomware Prevention Guide