A vulnerability scanner can report a clean, green dashboard while engines sit offline, check content is weeks old, scan credentials fail silently, exclusions grow unchecked and its own administrator and integration accounts are among the most powerful identities in the estate. This health check is a recurring, measurable review of the vulnerability scanning platform itself: whether its engines and agents work, whether its detection content is current and proven, whether scans run as designed, and who and what can reach the credentials and findings it holds. It turns those questions into thirteen metrics with formulas and thresholds so results can be compared quarter on quarter. It is written for vulnerability management, infrastructure security and audit teams looking for a "vulnerability scanner audit checklist" or "scanner health metrics".
This page measures the tool, not the programme. Remediation speed, SLA compliance, exceptions and overall scan coverage of the estate are measured in the Vulnerability Management Health Check, and none of its metrics are repeated here. Deployment, engine placement and secure credential design are covered in the Vulnerability scanner rollout playbook, and scanning practice in the Vulnerability Scanning Guide. The review is vendor-neutral and applies to network scanners, agent-based scanners and cloud-hosted scanning consoles.
What this health check covers
The review asks one question: if a critical, newly published vulnerability appeared on an in-scope host today, would the scanner have the content to detect it, an engine or agent able to reach the host, a credential that works, and no exclusion hiding it, and is the scanner itself a safe place to hold what it collects? It covers:
- Engine and agent health: scan engines and agents online, checking in and on a supported version.
- Detection content: how current the vulnerability checks are on every engine, and proof from a seeded test that they detect what they should.
- Scan execution: scheduled scans completing within their windows, reach into every network zone, and scans that disrupt services.
- Scan credentials: the privileged accounts the scanner uses to log in to targets, where they are stored and how they are rotated.
- The tool as a privileged system: exclusions, console roles, integration tokens and audit logging.
Who runs it: the scanner platform owner collects the data; the vulnerability management lead reviews; the CISO or head of security operations signs off. What it is not: a measure of how fast vulnerabilities are fixed or how much of the estate is covered by all assessment sources together. Those are programme outcomes.
Key takeaways
- Stale content is a silent failure. An engine that has not updated its checks in a week will not find last week's critical vulnerability, and the dashboard will not say so.
- Prove detection with a seeded reference host. A known, deliberately unpatched test system in each zone is the only end-to-end test that engines, credentials, content and reporting all work.
- Scan credentials are tier 0 secrets. They usually hold local administrator or root rights across most of the estate, so treat them like any other privileged non-human identity.
- Exclusions decay into blind spots. Every excluded host, range or check needs an owner, a reason and an expiry.
- The scanner is an attacker's map. Its findings list every weakness, so console roles, exports and integration tokens need the same control as a vault.
How to use this template
Copy the metrics table and the findings log into your own tracker, then run this vulnerability scanner health check at the cadence below. Collect each figure from the data source types described, score it against the thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review and its raw extracts so results can be compared quarter on quarter, and replace bracketed placeholders with your own values.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| Engine, agent and content update status | Weekly (automated) | Scanner platform owner | Vulnerability management lead | Engine and agent status export, content version per engine |
| Seeded reference host test | Monthly, and after content, engine or credential changes | Vulnerability analyst | Vulnerability management lead | Reference host list, expected and detected findings |
| Exclusion and scan policy review | Quarterly | Scanner platform owner | Vulnerability management lead | Exclusion export with owners and expiries, scan policy change history |
| Full metrics review (this page) | Quarterly | Scanner platform owner | CISO | Metrics table, raw extracts, findings log |
| Console role and integration credential recertification | Quarterly | Identity governance team | Vulnerability management lead | Role list, integration tokens and scan credential sets with reviewer decisions |
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Engine" means any scanning appliance, virtual scanner or cloud scanning node; "agent" means software installed on a host that reports to the scanner.
| ID | Metric | Formula | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| VS-01 | Engine availability | Deployed engines online and accepting jobs at each weekly check ÷ deployed engines, averaged over the quarter | Scanner console engine status export | At least 98% | 90% to below 98% | Below 90% |
| VS-02 | Agent check-in health | Installed agents that reported results within the last 7 days ÷ installed agents | Agent status export | At least 95% | 85% to below 95% | Below 85% |
| VS-03 | Detection content freshness | Engines and agents whose vulnerability check content was updated within the last 48 hours ÷ engines and agents | Content or plug-in version per engine and agent | At least 98% | 90% to below 98% | Below 90%, or any engine more than 7 days behind |
| VS-04 | Seeded detection rate | Expected findings on seeded reference hosts that the scanner reported in the test cycle ÷ expected findings | Reference host test plan, scan results for those hosts | At least 95% | 85% to below 95% | Below 85% |
| VS-05 | Scheduled scan completion | Scheduled scan jobs that finished all targets without abort or timeout ÷ scheduled scan jobs in the quarter | Scan job history | At least 97% | 90% to below 97% | Below 90% |
| VS-06 | Zone reach | In-scope network zones and cloud environments where a reachability test from the assigned engine succeeded this quarter ÷ in-scope zones and environments | Zone register, engine assignment, reachability test results | 100% | 90% to below 100% | Below 90% |
| VS-07 | Scan-induced disruption | Incidents or change records attributing a service impact to a scan in the quarter | Incident and change records tagged to scanning | 0 | 1 to 2, each with a tuned policy | More than 2, or any on a safety or OT system |
| VS-08 | Scan credential storage and rotation | Scan credential sets held in a vault or privileged access integration and rotated within policy ÷ scan credential sets configured | Scanner credential configuration export, vault rotation records | 100% | 90% to below 100% | Below 90%, or any domain-wide administrator credential stored only in the scanner |
| VS-09 | Scan credential scope | Scan credential sets limited to the target group they serve (no single credential valid across all zones) ÷ scan credential sets | Credential sets joined to target groups | 100% | 80% to below 100% | Below 80% |
| VS-10 | Exclusion hygiene | Excluded hosts, ranges and checks without owner, reason or expiry, or past expiry ÷ all exclusions | Scanner exclusion and scan policy export | 0% | Above 0% to 10% | Above 10%, or any whole-subnet exclusion without approval |
| VS-11 | Console privilege | People who can change scan policies, credentials or exclusions, or export all findings ÷ approved baseline for those roles | Console role assignment export | At or below baseline | Up to 20% above baseline | More than 20% above baseline, or any shared or local administrator account |
| VS-12 | Integration token hygiene | API keys, connector accounts and tokens (ticketing, CMDB, cloud, SIEM) unused for 90 days, past rotation or holding write rights they do not use ÷ integration credentials | Integration and API key inventory with last use | 0% | Above 0% to 10% | Above 10% |
| VS-13 | Platform patching and audit logging | Scanner components (console, engines, agents) on a supported version within your patching deadline and sending audit events that reached the SIEM in the last 7 days ÷ scanner components | Version inventory, SIEM ingestion report | At least 98% | 90% to below 98% | Below 90% |
VS-03 reflects NIST SP 800-53 Rev. 5 RA-5(2), which asks organisations to update the vulnerabilities to be scanned at a defined frequency. VS-08 and VS-09 reflect RA-5(5) Privileged Access, which covers privileged access authorisation for scanning activities, and IA-5 Authenticator Management.
How to collect the data
Use exports and tool telemetry, not questionnaires. Forward the scanner's audit log to the SIEM first so the figures can be rerun each quarter.
- Export engine and agent status weekly with online state, last check-in, software version and content version (VS-01 to VS-03, VS-13). A single quarterly snapshot hides engines that were offline for weeks.
- Build seeded reference hosts: one deliberately unpatched, isolated test system per major platform and zone, with a documented list of expected findings, authenticated and unauthenticated (VS-04). Rebuild them from a fixed image so results are comparable. Never seed production systems.
- Pull scan job history with start, end, targets attempted and completed, and abort reasons (VS-05). Jobs that "complete" after skipping unreachable targets count as incomplete.
- Test reach: from each engine, confirm it can reach a known host in every zone it is assigned to, and that firewall rules permit the scan (VS-06). Record zones with no assigned engine as failures.
- Query incident and change records for service impacts attributed to scanning (VS-07), especially on fragile, legacy and OT systems. See Safe active scanning and testing in OT.
- Export credential sets with type, storage method, last rotation and target groups, and match them to vault rotation records (VS-08, VS-09). The Authenticated vs Unauthenticated Scanning Guide covers credential design.
- Export exclusions (hosts, ranges, checks and scan windows) with author, date, reason and expiry (VS-10).
- Export console roles and integrations: every role that can change policy, credentials or exclusions or export findings, and every API key and connector with scope and last use (VS-11, VS-12).
Red flags that need action now
- An engine whose detection content has not updated for more than 7 days, or engines pinned to an old content version with no change record.
- A seeded reference host where a known critical vulnerability was not reported.
- A domain-wide or organisation-wide administrator account used as the scan credential, stored only in the scanner.
- Repeated authentication failures on scan credentials across one zone, which may mean the credential was changed outside the process or is being locked out by an attacker's guessing.
- Scan credentials used to log in outside scan windows or from hosts that are not scan engines.
- A whole subnet, a cloud account or a high-severity check excluded with no owner or expiry.
- New console administrators, new integration tokens or bulk exports of findings with no change record.
- Scanner console or engines running a version past the vendor's supported release.
Scan credential misuse maps to ATT&CK Valid Accounts: Domain Accounts (T1078.002). If a red flag suggests the scanner itself has been tampered with, follow If your vulnerability scanner is compromised rather than logging a finding.
Findings log
Log each amber or red result and each red flag. Carry open items into the next quarter.
| ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-VSCAN-01 | [Description, e.g. 3 engines in the data centre zone 12 days behind on content] | VS-03 | [High] | [Scanner platform owner] | [Date] | [Open] |
| HC-VSCAN-02 | [Description, e.g. one Windows scan credential valid across every zone] | VS-09 | [High] | [Identity team] | [Date] | [Open] |
| HC-VSCAN-03 | [Description] | [ID] | [Critical / High / Medium / Low] | [Role] | [Date] | [Open / In progress / Closed / Risk accepted] |
Suggested due dates: critical within 7 days, high within 30 days, medium within 90 days.
Remediation map
| Finding type | Where to fix it |
|---|---|
| Engines offline, agents not checking in, content stale | Vulnerability scanner rollout playbook (architecture and placement) |
| Missed seeded findings, unreachable zones | Vulnerability Scanning Guide |
| Failing or over-broad scan credentials | Authenticated vs Unauthenticated Scanning Guide |
| Scan credentials outside a vault or not rotated | Credential Rotation at Scale |
| Scan disruption on fragile systems | Safe active scanning and testing in OT |
| Exclusions without owners or expiry | Remediation SLAs and Vulnerability Exceptions Guide |
| Console roles, integration tokens, audit logging | Vulnerability scanner rollout playbook (securing the platform) |
| Suspected tampering | If your vulnerability scanner is compromised |
Framework mapping
| Area | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | CIS Controls v8.1 |
|---|---|---|---|
| Engines, agents and content (VS-01 to 04) | ID.RA-01 | RA-5, RA-5(2), RA-5(3) | Control 7 |
| Scan execution and reach (VS-05 to 07) | ID.RA-01, PR.PS-01 | RA-5, CM-6 | Control 7 |
| Scan credentials (VS-08, 09) | PR.AA-01, PR.AA-05 | RA-5(5), IA-5, AC-6 | Control 5, Control 7 |
| Exclusions, roles, tokens, logging (VS-10 to 13) | PR.AA-05, PR.PS-04 | AC-6, AU-2, AU-9 | Control 6, Control 8 |
Related guides
- Vulnerability scanner rollout playbook: the deployment and hardening this review tests.
- Vulnerability Management Health Check: programme outcomes such as remediation time, SLA compliance and estate coverage.
- Vulnerability Management Maturity Model: place the results on a maturity level.
- Vulnerability Management Metrics: board-level programme reporting.
- If your vulnerability scanner is compromised: when a red flag is an incident.
References
- NIST: SP 800-53 Rev. 5, Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-115, Technical Guide to Information Security Testing and Assessment
- NIST: SP 800-40 Rev. 4, Guide to Enterprise Patch Management Planning
- CIS: Critical Security Controls v8.1
- MITRE ATT&CK: Valid Accounts: Domain Accounts (T1078.002)
Related NHI Mgmt Group resources: Vulnerability scanner rollout playbook, Vulnerability Scanning Guide, Vulnerability Management Health Check, Authenticated vs Unauthenticated Scanning Guide.