A secrets scanner gives a comforting dashboard while new repositories go unscanned, push-time blocking is switched off for a deadline, internal token formats never match a rule and dismissed findings quietly include live credentials. This health check is a recurring, measurable review of a secrets scanning tool already in production: what it scans, whether its detectors still find what you issue, how findings move through triage, and who and what can reach the secrets it has collected. It turns those questions into thirteen metrics with formulas and thresholds, so the result can be compared quarter on quarter. It suits AppSec, platform and NHI teams looking for a "secret scanning audit checklist" or "secret scanning KPIs".
This page measures the tool; it does not explain how to choose, deploy or protect it. For roles, the scanner's own credentials, finding protection and CI placement, use Securing the secrets scanning tool itself. For detector coverage by credential type and how to test it, use the Secret Scanner Rule Coverage Comparison Guide, and for scanning practice in repositories, the Git Secret Scanning Guide. The review is vendor-neutral and applies to platform-native scanning, standalone scanners and open source detectors run in pipelines.
What this health check covers
The review asks: if a developer, pipeline or AI coding assistant committed a live credential today, would the tool see it, block it or route it to someone who acts, and is the tool itself a safe place to hold what it finds? It covers:
- Surface coverage: repositories, full history, container images, CI logs and other surfaces such as wikis, tickets and chat, measured against inventories.
- Prevention: push-time and pre-receive blocking, and the bypass rate.
- Detector health: scan job success, custom detectors for your own credential formats, and seeded test results.
- Finding workflow inside the tool: triage time, backlog age, owner routing and the quality of dismissals.
- The tool as a credential store: who can see unmasked values, allowlist and exclusion hygiene, and the scanner's own tokens and integrations.
Who runs it: the AppSec or secrets scanning owner collects the data; the head of application or platform security reviews; the CISO signs off. What it is not: a measure of how fast the organisation rotates secrets across the estate or how many secrets live in vaults. Those are secrets management programme outcomes and are measured separately, so the metrics below stay with the tool.
Key takeaways
- Coverage is measured against inventories. Count scanned repositories against the source control platform's list of active repositories, not against what the scanner already knows about.
- Prevention beats detection. Push-time blocking keeps a secret out of history; track both its coverage and how often developers bypass it.
- Your own token formats are the blind spot. Generic patterns and partner formats rarely match internal API keys and service tokens, so count custom detectors against the formats you issue.
- Test with seeded canaries. Planting inert test credentials of each type is the only proof the detectors, pipelines and alert routing still work end to end.
- The scanner is a credential store. It holds every secret it has found; unmasked viewers, exports and its integration tokens need the same control as a vault.
How to use this template
Copy the metrics table and the findings log into your own tracker, then run this secrets scanning tool health check at the cadence below. Collect each figure from the data source types described, score it against the thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review and its raw extracts so results can be compared quarter on quarter.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| New repository and surface reconciliation | Weekly (automated) | Secrets scanning owner | AppSec lead | Repository inventory joined to scanner coverage, onboarding tickets |
| Dismissal sample and bypass review | Monthly | AppSec engineer | AppSec lead | Sampled dismissals with verdicts, bypass events with reasons |
| Seeded canary test | Quarterly, and after detector or pipeline changes | AppSec engineer | AppSec lead | Test plan, canary types planted, detections and alerts received |
| Full metrics review (this page) | Quarterly | Secrets scanning owner | CISO | Metrics table, raw extracts, findings log |
| Role and integration credential recertification | Quarterly | Identity governance team | AppSec lead | Role list, unmasked viewers, integration tokens with reviewer decisions |
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Active repository" means any repository with a commit, pull request or pipeline run in the last 180 days. "Verified live" means the tool or a validity check confirmed the credential still works.
| ID | Metric | Formula | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| SS-01 | Repository coverage | Active repositories with scanning enabled and a completed full history scan ÷ active repositories in the source control inventory | Source control platform repository list, scanner coverage export | At least 98% | 90% to below 98% | Below 90% |
| SS-02 | Push-time blocking coverage | Active repositories with push protection or a pre-receive secret check enforced ÷ active repositories | Scanner or platform push protection settings, pre-receive hook configuration | At least 95% | 75% to below 95% | Below 75% |
| SS-03 | Push-time bypass rate | Blocked pushes that were bypassed ÷ pushes blocked | Push protection bypass events with reasons | Below 5%, all with a reason | 5% to 15% | Above 15%, or bypasses with no reason |
| SS-04 | Non-code surface coverage | In-scope non-code surfaces (container registries, CI logs and artefacts, wikis, ticketing, chat) connected and scanned in the last 7 days ÷ in-scope surfaces | Surface register, scanner integration status and last scan dates | At least 90% | 70% to below 90% | Below 70% |
| SS-05 | Scan job success | Scheduled and pipeline scan jobs that completed without error or timeout ÷ scan jobs started | Scanner job history, CI job logs | At least 99% | 95% to below 99% | Below 95% |
| SS-06 | Internal format detector coverage | Internal credential formats (API keys, service tokens, signing keys you issue) with a tested custom detector ÷ internal formats in the credential format register | Credential format register, custom detector list | 100% | 80% to below 100% | Below 80% |
| SS-07 | Seeded canary detection rate | Inert test credentials planted across surfaces and types that produced an alert routed to the right team within 1 hour ÷ canaries planted | Canary test plan, scanner alert log, ticketing or chat receipts | At least 95% | 80% to below 95% | Below 80% |
| SS-08 | Verified-live triage time | Verified-live findings triaged (classified and assigned) within 4 hours of detection ÷ verified-live findings | Finding timestamps, triage and assignment history | At least 95% | 80% to below 95% | Below 80% |
| SS-09 | Open finding backlog age | Open findings older than 30 days ÷ open findings | Finding export with created date and state | Below 5% | 5% to 20% | Above 20% |
| SS-10 | Owner routing | New findings automatically routed to an owning team through repository or surface ownership data ÷ new findings | Finding assignment data, repository ownership metadata | At least 95% | 80% to below 95% | Below 80% |
| SS-11 | Dismissal quality | Sampled dismissals with a recorded reason and confirmed correct on review ÷ dismissals sampled (minimum 25 or all, if fewer) | Dismissal records, reviewer verdicts | At least 95% | 85% to below 95% | Below 85%, or any verified-live secret dismissed as false positive |
| SS-12 | Exclusion hygiene | Path, repository, file type and pattern exclusions without owner, justification or expiry, or past expiry ÷ all exclusions | Scanner exclusion and allowlist export, inline ignore annotations | 0% | Above 0% to 10% | Above 10%, or any exclusion of a whole repository or configuration file type |
| SS-13 | Unmasked access and integration credentials | People with rights to view unmasked secret values or export findings, plus scanner integration tokens unused for 90 days or past rotation ÷ the approved baseline for each | Role assignment export, integration and token inventory | At or below baseline, all tokens in date | Up to 20% above baseline | More than 20% above baseline, or any long-lived token with write scope unused for 90 days |
Report SS-01, SS-02 and SS-07 per business unit or source control organisation, and separately for repositories that hold infrastructure as code, which carry the most cloud credentials.
How to collect the data
Use exports and tool telemetry rather than questionnaires. Where the scanner offers an audit log, forward it to the SIEM first so these figures can be rerun each quarter.
- Fix the denominators. Export the active repository list from every source control organisation and instance, including archived but readable ones, and the register of non-code surfaces in scope (SS-01, SS-04). Mergers, shadow organisations and personal namespaces are where coverage gaps hide.
- Export scanner coverage and settings: per repository, whether scanning, history scanning and push-time blocking are on, and the last scan date (SS-01, SS-02). Include pre-receive hooks and developer-side hooks; the OWASP Secrets Management Cheat Sheet recommends detection before commit, but treat local hooks as advisory because developers can skip them.
- Pull bypass events with user, repository, reason and whether the secret was later confirmed live (SS-03).
- Read the job history for scheduled scans and pipeline scan steps, counting errors, timeouts and steps marked as allowed to fail (SS-05). A scan step that is allowed to fail is often failing silently.
- Keep a credential format register of every token type your organisation issues, with its prefix or structure, and match it to custom detectors (SS-06). The Secret Scanner Rule Coverage Comparison Guide covers how to design detectors and spot checks.
- Run the canary test. Plant inert, clearly labelled test credentials of each type in a test repository, a CI log, a container image layer and a ticket, then record which raised an alert and where it was routed (SS-07). Never use live credentials for this.
- Export findings with created, triaged, assigned, dismissed and closed timestamps, validity status and owner (SS-08 to SS-10). The Developer Secret Scanning Triage guide sets what good triage looks like.
- Sample dismissals randomly, with a second reviewer checking reason and correctness (SS-11). Export exclusions, allowlists and inline ignore annotations with authors and dates (SS-12).
- Export access and integrations: roles that can reveal full values or export findings, and every token, app installation and service account the scanner uses, with scopes and last use (SS-13).
Red flags that need action now
- A verified-live finding dismissed as a false positive or "used in tests" with no rotation.
- Push-time blocking turned off for a whole organisation or a group of infrastructure repositories.
- A cluster of bypasses from one team or one automation account.
- Canary credentials planted in CI logs or container images that produced no alert.
- An exclusion for a whole repository, a configuration file type or a directory such as test fixtures that real code also uses.
- Scan steps in pipelines set to continue on error for months.
- Many people able to reveal full secret values, or bulk finding exports with no business reason.
- A scanner integration token with write access to source control that has never been rotated.
Findings log
Log each amber or red result and each red flag. Carry open items into the next quarter.
| ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-SECSCAN-01 | [Description, e.g. 212 repositories in an acquired organisation with no scanning] | SS-01 | [High] | [Secrets scanning owner] | [Date] | [Open] |
| HC-SECSCAN-02 | [Description, e.g. internal service token format has no custom detector] | SS-06 | [High] | [AppSec engineer] | [Date] | [Open] |
| HC-SECSCAN-03 | [Description] | [ID] | [Critical / High / Medium / Low] | [Role] | [Date] | [Open / In progress / Closed / Risk accepted] |
Suggested due dates: critical within 7 days, high within 30 days, medium within 90 days. A verified-live secret found during the review is an incident, not a finding: follow the Leaked Credential Response Playbook.
Remediation map
| Finding type | Where to fix it |
|---|---|
| Repository and history coverage gaps | Git Secret Scanning Guide |
| Push-time blocking off or frequently bypassed | Git Secret Scanning Guide (pre-receive and push-time controls), Securing the secrets scanning tool itself (the scanner inside CI/CD) |
| Container image and build artefact gaps | Secrets in Container Images and Build Artefacts Guide |
| Missing detectors, failed canaries | Secret Scanner Rule Coverage Comparison Guide |
| Slow triage, backlog, poor dismissals | Developer Secret Scanning Triage |
| Broad exclusions, unmasked viewers, scanner tokens | Securing the secrets scanning tool itself |
| Suspected tampering with the scanner | If your secrets scanning tool is compromised |
| Repeated capability gaps | Secrets Scanning Buyer's Guide |
Framework mapping
| Area | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | CIS Controls v8.1 | MITRE ATT&CK |
|---|---|---|---|---|
| Coverage and prevention (SS-01 to 04) | PR.PS-06, ID.RA-01 | IA-5, RA-5 | Control 16 | Credentials In Files (T1552.001) |
| Detector and job health (SS-05 to 07) | DE.CM-09 | SI-4 | Control 16 | Credentials In Files (T1552.001) |
| Finding workflow (SS-08 to 11) | DE.AE-02, RS.MI-01 | IR-4, IA-5 | Control 16, Control 17 | |
| Exclusions, access and integrations (SS-12, 13) | PR.AA-05, PR.PS-01 | AC-6, CM-6 | Control 5, Control 6 | Cloud Accounts (T1078.004) |
Related guides
- Securing the secrets scanning tool itself: the hardening that SS-12 and SS-13 test.
- Secrets Management Guide: where the credentials the scanner finds should live instead.
- Secrets Management Maturity Model: place scanning results on a maturity level.
- AI Coding Tools and Secrets Sprawl: a growing source of the commits this tool must catch.
- If your secrets scanning tool is compromised: when a red flag is an incident.
References
- OWASP: Secrets Management Cheat Sheet
- NIST: SP 800-53 Rev. 5, Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- CIS: Critical Security Controls v8.1
- MITRE ATT&CK: Unsecured Credentials: Credentials In Files (T1552.001)
Related NHI Mgmt Group resources: Securing the secrets scanning tool itself, Secret Scanner Rule Coverage Comparison Guide, Git Secret Scanning Guide, Secrets Management Guide.