AppSec Programme Metrics Guide: Defining MTTR, Escape Rate, Coverage and Debt

Most application security dashboards count findings, and finding counts tell leadership almost nothing about whether software is getting safer. A rising count can mean better scanning; a falling one can mean a scanner was switched off. Defensible AppSec metrics need precise definitions, a trustworthy denominator and a clear link to a decision. This guide defines the core programme metrics (coverage, mean and median time to remediate, escape rate, security debt, SLA adherence and recurrence), shows how to calculate them without the usual statistical traps, sets out targets by severity, and maps them to OWASP SAMM, NIST SSDF, NIST SP 800-55 and CSF 2.0. It is for AppSec leads, product security managers and CISOs who report on software risk. Tooling that aggregates findings is covered in the Application Security Posture Management (ASPM) Guide; this page is about what to measure and how.

Key takeaways

• Coverage comes first: every other metric is only as good as its denominator, so measure the share of in-scope applications and repositories that each control actually reaches before reporting outcomes.
• Report median and percentile time to remediate, including open items: a mean over closed findings hides the oldest, riskiest backlog and rewards closing easy issues.
• Escape rate measures whether shift-left works: the share of security defects first found in production (by pentest, bug bounty, incident or customer) is the clearest outcome metric for a secure development life cycle.
• Debt is age times severity, not count: track SLA-breached and age-weighted backlog per product, and treat accepted risks as debt with an expiry date.
• Every metric needs an owner and a decision: if no one would act differently when a number moves, remove it from the report.

Why AppSec metrics go wrong

NIST SP 800-55 Volume 1 (Measurement Guide for Information Security, final December 2024) frames measurement as a way to judge the adequacy of policies, procedures and controls, and to prioritise and evaluate measures rather than collect them. OWASP SAMM's Strategy and Metrics practice sets the same progression: at level 1 define metrics that show the effectiveness and efficiency of the programme, at level 2 set targets and KPIs, and at level 3 use metrics and organisational needs to influence strategy. Programmes that skip to dashboards fail in predictable ways:

• Tool output as the metric: "12,400 SAST findings" depends on rules enabled, not on risk. Changing a query suite moves the number more than any engineering work does.
• No denominator: "95% of scanned repositories pass" says nothing if only half the repositories are scanned.
• Survivorship bias: time-to-fix computed only on closed findings ignores everything still open.
• Mixed populations: blending a dependency CVE with a design flaw from a threat model produces averages that describe neither.
• Gaming: when teams are judged on counts, findings get dismissed, downgraded or split. Pair every speed metric with a quality control on dismissals.

The core metric set

A workable programme reports a small set of metrics across four families. Definitions must be written down and versioned so that a change in formula is not mistaken for a change in risk.

Measuring time to remediate properly

MTTR is the most reported and most abused AppSec metric. Make these choices explicit:

• Clock start: first detection by any tool for that finding fingerprint, not ticket creation. If tickets lag detection by weeks, measure that lag separately (triage time).
• Clock stop: fix verified by a rescan or test in the environment where the risk lived (for production findings, the deployed fix), not "merged" or "ticket closed".
• Median and P90, not mean alone: remediation times are heavily skewed. A mean of 40 days can hide a median of 6 and a tail of 300. Report median for typical performance and P90 for the tail.
• Include open findings: open items are "censored" data: their fix time is at least their current age. Either report the age distribution of open findings alongside closed-item MTTR, or use survival analysis (the share of findings still open after 7, 30, 90 days), which handles open and closed items together.
• Segment: by severity, by source (SAST, SCA, DAST, pentest, bug bounty), by product tier and by fix type. A dependency upgrade and a redesign should never share an average.
• Handle reopen and duplicates: deduplicate by fingerprint before counting, and restart nothing when a finding reopens: keep the original start date so regressions show up.

Suggested remediation targets

Targets are an organisational risk decision; the values below are common starting points to calibrate against your own median times, not a standard.

Severity should reflect exploitability and context, not CVSS base score alone: FIRST's EPSS gives each CVE a daily probability (0 to 1) of exploitation activity in the next 30 days, and the CISA KEV catalog lists vulnerabilities with known exploitation. Using both lets SLAs follow real risk. Prioritisation models are covered in the Application Security Posture Management (ASPM) Guide.

Escape rate and defect origin

Escape rate answers the question leadership actually asks: do our development controls stop vulnerabilities before customers are exposed? Define it per product and per quarter:

escape rate = security defects first found post-release / all security defects found (pre-release + post-release)

• Pre-release sources: design review and threat modelling, IDE and pull request scanning, CI SAST, SCA and secrets scanning, pre-release DAST and pentest.
• Post-release sources: production DAST, scheduled pentests of live systems, bug bounty and vulnerability disclosure reports, incidents, customer reports.
• Weight by severity: report escape rate separately for critical and high defects; ten escaped low-severity issues matter less than one escaped authentication bypass.
• Record defect origin: for each escaped defect, record the life cycle phase where it should have been caught and the CWE. This turns escape rate into an improvement backlog (missing rule, missing test, missing threat model).
• Read it with coverage: a low escape rate with low post-release testing coverage means you are not looking, not that you are good. Track bug bounty and pentest coverage alongside it.

External reports are a source of escapes only if they can reach you. CISA's Secure by Design pledge asks signatories to publish a vulnerability disclosure policy with a reporting channel, and NIST CSF 2.0 ID.RA-08 calls for processes to receive, analyse and respond to vulnerability disclosures. The Home Depot token exposure shows the cost of having no intake: a GitHub token with access to private repositories and pipelines was exposed for more than a year, the researcher's emails went unanswered because there was no disclosure programme, and it was revoked only after a journalist contacted the company. Measure time from external report to acknowledgement and to fix as part of the speed family.

Coverage: building a trustworthy denominator

1. Define in scope: every repository with production deployments and every externally reachable application, including third-party built and low-code applications you operate.
2. Reconcile sources: join the source control inventory, CI/CD pipelines, cloud asset inventory and DNS. Applications found in cloud or DNS but not in source control are unowned risk.
3. Assign owner and tier: no owner means no one to send metrics to. Tier drives SLA and which controls are mandatory.
4. Measure "active and producing results": a scanner configured but failing every run is not coverage. Count a control only if it completed successfully in the period (for example, a SAST analysis in the last 7 days on the default branch).
5. Report coverage by control and tier: for example, tier 1 SAST 98%, SCA 100%, threat model 60%, annual pentest 85%.

Security debt and exception health

Security debt is the backlog of known, unremediated weaknesses. Counting it is misleading because a thousand informational findings outweigh one critical. A simple, explainable formula is a severity-weighted, age-weighted sum of findings past their SLA:

debt = sum over overdue findings of (severity weight x days overdue), with weights such as critical 10, high 5, medium 2, low 0.5.

• Trend it per product: the absolute number matters less than whether debt grows or shrinks quarter to quarter, and which products carry most of it (typically a small minority).
• Accepted risk is debt with a date: each exception should have an owner, a compensating control, an approver at the right level, and an expiry date. Report expired exceptions as overdue findings.
• Separate design debt: architectural weaknesses from threat models (missing authorisation layer, shared credentials) need roadmap items, not tickets. Track them as a separate list with planned quarters.
• Recurrence shows root causes: repeated CWE classes in one product point to missing secure defaults or libraries. CISA's Secure by Design pledge asks manufacturers to show measurable reductions in entire classes of vulnerability (such as SQL injection, cross-site scripting or memory safety), which is a useful framing for this metric.

Connecting security to delivery metrics

Security metrics carry more weight with engineering when they sit beside the delivery metrics teams already use. DORA defines five software delivery metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. Three practical links:

• Lead time for security fixes: measure change lead time for security-labelled changes separately. If it is much longer than for features, the problem is process (approvals, release trains), not engineering skill.
• Gate cost: track how often security gates block a pull request and how long blocked changes wait. Rising block rates with stable escape rates mean gates are tuned too tight.
• Security-caused failures: count failed deployments and rework caused by security changes; a high figure argues for better test environments, not fewer controls.

Reporting by audience

For executive reporting formats see the Vulnerability Remediation Report Template for Executives, and for the wider security metrics programme the Security Metrics and KPIs Programme Guide.

How to implement

1. Write the metric catalogue: for each metric record definition, formula, data source, owner, cadence, target and the decision it informs. Version it.
2. Fix the inventory: reconcile repositories, applications and owners before publishing any outcome metric.
3. Normalise findings: deduplicate across tools by fingerprint, map every finding to severity, CWE, product and tier. An ASPM platform or a data warehouse can do this.
4. Baseline for one quarter: publish internally without targets, so teams can challenge definitions and data quality.
5. Set targets and SLAs: base them on the baseline median and P90, then tighten annually.
6. Pilot reporting with two business units: check that the numbers lead to decisions (funding, sprint priorities, gate tuning) before scaling.
7. Review quarterly: retire metrics nobody uses, and record formula changes so trends are not misread. Roll back a metric change if it breaks comparability without a clear benefit.

For small organisations, four metrics are enough: coverage of SCA and secrets scanning, critical and high SLA adherence, age of the oldest open critical, and the number of externally reported issues. Large organisations need the full set, segmented by business unit and tier.

Where identity fits

Leaked credentials are a finding class that standard AppSec metrics often miss. Add secrets exposure to the metric set: time from a secret being committed or exposed to it being revoked (not just removed from code), and the share of detected secrets that were rotated. A secret deleted from a repository but still valid is open risk. See the Identity Security Metrics Guide for the identity side of measurement.

Common failure patterns

• Celebrating a falling count after a tool change: a scanner upgrade or rule change is recorded as improvement. Annotate trend charts with tooling changes.
• MTTR from closed items only: teams close fresh, easy findings first and MTTR improves while the critical backlog ages.
• Dismissal as remediation: "won't fix" and "false positive" count as fixed. Report them separately and review a sample every month.
• Coverage counted by configuration: repositories with a scanner configured but failing for months counted as covered.
• Secrets removed but not revoked: exposure metrics stop when the commit is cleaned, while the token stays valid, as in the Home Depot case above.
• League tables: ranking teams by finding counts pushes findings into private backlogs or off the scanner altogether.

Framework mapping

Practitioner checklist

• Write and version a metric catalogue with definition, formula, source, owner, target and decision for each metric.
• Reconcile the application inventory and owners before reporting outcomes; report inventory completeness.
• Count coverage only where a control completed successfully in the period, per control and tier.
• Report median and P90 time to remediate per severity and source, plus the age of open findings or survival curves.
• Measure escape rate per product with defect origin and CWE recorded for every escaped defect.
• Track severity and age weighted security debt and expiring exceptions per product.
• Use KEV and EPSS to define an exploitable exposure metric for internet-facing services.
• Report dismissals and false positives separately and review samples monthly.
• Add secrets exposure-to-revocation time to the metric set.
• Annotate trends with tooling and formula changes; retire metrics that drive no decision.

Standards and references

• OWASP SAMM: Strategy and Metrics
• NIST: SP 800-55 Vol. 1, Measurement Guide for Information Security
• NIST: SP 800-218, Secure Software Development Framework (SSDF) Version 1.1
• NIST: Cybersecurity Framework 2.0
• CISA: Secure by Design Pledge
• CISA: Known Exploited Vulnerabilities Catalog
• FIRST: Exploit Prediction Scoring System (EPSS)
• DORA: Software delivery performance metrics

Related NHI Mgmt Group resources: Application Security Posture Management (ASPM) Guide · AppSec Maturity Model · Security Metrics and KPIs Programme Guide · SAST Triage and False Positive Management Guide