Most certificate outages are not caused by a missing certificate but by a renewed one that never reached the system serving it. This template gives [Organisation] a per-certificate renewal runbook: who owns each certificate, which services depend on it, the renewal and deployment steps, how to validate the certificate actually served and what to do when renewal fails or a certificate has already expired. It is written for platform, PKI and site reliability teams preparing for publicly trusted TLS lifetimes falling to 100 days in March 2027 and 47 days in March 2029. Fill in one copy per certificate or per certificate family, and keep the completed runbooks alongside your certificate inventory. For why expiry outages happen and the full CA/Browser Forum schedule, see the Certificate Expiry Outages and 47-Day TLS Certificates Guide.
How to use this template
- One runbook per certificate family: group certificates that share an owner, issuer and deployment path (for example "public web front ends behind [load balancer]"). Where a certificate has unusual targets, such as an appliance, a Java keystore or a pinned mobile client, give it its own runbook.
- Two modes: sections 3 to 6 cover both planned renewal (the normal path, ideally fully automated) and the incident path (an expired or about-to-expire certificate, or a CA-driven mass revocation). The incident path follows the same headings as an incident response plan so it plugs into [Organisation]'s existing process.
- Automate, then document the exceptions: where an ACME client or a managed service renews and deploys without people, this runbook records the configuration, the monitoring and the manual fallback. Where renewal is still manual, it records every step so that someone other than the usual owner can do it.
- Keep it short enough to use: replace the guidance text in square brackets, delete rows that do not apply and keep the evidence column. The post-incident review and auditors will ask for it.
- Owner and review: [Head of Platform or PKI Owner], reviewed every [6] months, after each failed renewal and whenever a deployment target, issuing CA or ACME client changes.
1. Scope and triggers
1.1 This runbook covers the certificates listed in the table below. Code-signing certificates, S/MIME certificates and CA certificates are out of scope; use the Compromised Code Signing Certificate Playbook and the [CA operations procedure].
| Field | Value for this certificate |
|---|---|
| Certificate name or common name and SANs | [www.example.com, api.example.com] |
| Inventory ID and serial number of current certificate | [CLM record ID, serial] |
| Issuer and type | [Public CA name, private CA name]; [public TLS, private TLS, mTLS client, device] |
| Validity period and current expiry | [e.g. 90 days, expires YYYY-MM-DD] |
| Renewal method | [ACME client and version, cloud managed renewal, CLM platform, manual CSR] |
| Domain validation method | [HTTP-01, DNS-01, DNS account-labelled or persistent TXT, not applicable for private CA] |
| Private key location and protection | [file path and permissions, HSM or cloud KMS key ID, secrets manager path] |
| Deployment targets | [every host, load balancer, CDN, API gateway, keystore and appliance that serves or trusts it] |
| Dependent services and their criticality | [service name, tier from the business impact analysis] |
1.2 Planned renewal triggers. Renewal starts when any of the following occurs: the ACME client's renewal check fires (for example, from Certbot 4.0.0 a certificate is due when less than one third of its lifetime remains); the CA's ACME Renewal Information (ARI) suggested window opens; or the certificate reaches [one third] of its remaining lifetime for manual certificates. Let's Encrypt advises renewing at about two thirds of the validity period and warns that fixed intervals such as 60 days will not work with 45-day certificates.
1.3 Incident triggers. The incident path opens when any of the following is confirmed by the Certificate Owner or on-call engineer:
| Trigger | Typical evidence | Severity |
|---|---|---|
| Certificate expired in production | Client errors such as certificate expired, synthetic check failure, spike in TLS handshake failures | [Sev 1 if a Tier 1 service is affected] |
| Certificate inside [7] days of expiry with automated renewal failing | ACME client errors, ARI window passed, alert from expiry monitoring | [Sev 2] |
| Renewed certificate not served | Inventory shows new serial, external scan shows old serial on one or more endpoints | [Sev 2] |
| CA revocation notice or mass revocation event | CA email or problem report, ARI window moved forward. The Baseline Requirements oblige CAs to revoke within 24 hours or 5 days depending on the reason | [Sev 1 or 2 by number of certificates] |
| Private key suspected exposed | Key found in a repository, backup or ticket; host compromise | [Sev 1, also open the security incident process] |
2. Roles and contacts
| Role | Named owner and deputy | Responsibilities in this runbook |
|---|---|---|
| Certificate Owner | [Name, team, deputy] | Accountable for this certificate's renewal, approves emergency issuance, keeps the runbook current |
| Service Owner (one per dependent service) | [Name per service] | Approves deployment windows, confirms service health after deployment |
| On-call platform engineer | [Rota name, paging route] | Responds to expiry and renewal-failure alerts, runs sections 3 to 6 |
| PKI or CLM administrator | [Name, deputy] | Operates the CLM platform, private CA or ACME account, issues manually if automation fails |
| DNS administrator | [Name, deputy] | Fixes DNS-01 or persistent validation records, CAA records |
| CA account manager or support | [CA name, support URL, account ID, contract number] | Expedited issuance, revocation questions, validation problems |
| Incident Manager | [Rota] | Takes command for Sev 1, owns communications and the decision log |
2.1 Renewal notices and alerts go to a team distribution list or alerting route, never to an individual mailbox. Let's Encrypt stopped sending expiry notification emails in 2025, so CA email cannot be the only warning.
2.2 The ACME account key, CA portal credentials and any API tokens used by the renewal pipeline are non-human identities. Record where each is stored, who can use it and when it was last rotated: [vault path, owner, rotation date]. See the Machine Identity, PKI and Certificate Lifecycle Guide.
3. Detection and triage
3.1 Monitoring in place. Record the checks that would detect a problem with this certificate, and test each one:
- Served-certificate check: an external or synthetic probe connects to every endpoint in 1.1, reads the presented certificate and alerts at [30], [14] and [7] days before expiry, and when the presented serial differs from the inventory serial.
- Renewal job health: the ACME client or CLM platform reports success or failure of each renewal attempt to [monitoring system]; a missed run is an alert, not only a failed one.
- ARI polling: where the CA supports RFC 9773, the client fetches the renewalInfo resource, schedules renewal at a random time inside the suggestedWindow and alerts if the window passes without a new certificate.
- Certificate Transparency: for public certificates, monitor CT logs for unexpected issuance on [domains]. See the Certificate Transparency Monitoring Guide for Web Teams.
3.2 Triage questions for the on-call engineer, answered in the incident record within [15] minutes:
- Which certificate (serial and SANs) and which endpoints are affected? Is the problem the leaf, an intermediate or the client's trust store?
- Has a valid replacement already been issued? If so, the problem is deployment or reload, go to 6.2.
- If not, why did issuance fail: validation (DNS or HTTP challenge), CAA record, rate limit, ACME account or credential, CA outage, or a policy block in the CLM platform?
- Is there any suggestion of key compromise? If yes, do not reuse the key, and open the security incident process in parallel.
- How many other certificates share the same renewal path and could fail the same way?
4. Containment
4.1 The aim of containment is to stop the outage spreading and buy time, without weakening trust:
- Freeze changes on the affected deployment path ([load balancer, CDN, ingress]) except those needed for the fix.
- Fail over to an endpoint that still presents a valid certificate where one exists, for example another region or CDN configuration: [failover procedure reference].
- Do not ask users or partners to bypass certificate errors and do not disable certificate validation in clients or service-to-service calls. Record any request to do so in the decision log and refuse it unless the Incident Manager and [CISO] approve a time-boxed exception.
- Stop automated retries that are hitting CA rate limits until the root cause in 3.2 is fixed.
5. Eradication
5.1 Fix the cause of the failure before issuing again, using the table below.
| Cause | Fix | Evidence |
|---|---|---|
| DNS-01 or HTTP-01 challenge failing | Restore DNS API credentials or challenge path; check proxies and redirects; confirm CAA allows the CA (and, from 15 March 2027, that any accounturi or validationmethods parameters match) | [Challenge log, DNS change ID] |
| Renewal job not running | Restore the scheduler, container or service account; add missed-run alerting | [Job history] |
| ACME account or CA credential invalid | Rotate the credential through [vault]; never paste it into a ticket | [Rotation record] |
| Deployment hook failed | Fix the hook; with Certbot use a deploy hook (runs only after successful renewal), noting a failing hook does not stop renewal itself | [Hook output] |
| Wrong renewal threshold | Replace fixed day counts with a fraction of lifetime or ARI; in cert-manager prefer spec.renewBeforePercentage over a fixed renewBefore | [Config diff] |
| Suspected key compromise | Generate a new key pair in [HSM or KMS], request revocation of the old certificate, follow the security incident process | [Revocation confirmation] |
6. Recovery
6.1 Issue the replacement. Prefer the normal automated path once fixed. If it cannot be fixed in time, the PKI or CLM administrator issues manually:
- Generate a new key pair where the key cannot be confirmed safe; otherwise follow the key reuse rule in [Certificate Management Policy section].
- Request the certificate with the same SANs; do not request the maximum validity. The Baseline Requirements say certificates should not exceed 199 days from March 2026, 99 days from March 2027 and 46 days from March 2029.
- Where the ACME server supports ARI, send the replaces field identifying the predecessor certificate so the CA can link the renewal.
- Check the full chain returned by the CA and that the intermediates are the ones the clients trust.
6.2 Deploy and reload. Complete one row per deployment target. A certificate on disk is not deployed until the process serving it has reloaded.
| Target | Install step | Reload or restart | Validated by |
|---|---|---|---|
| [Web server host group] | [path for cert, chain and key; permissions] | [graceful reload command reference] | [name, time] |
| [Load balancer or CDN] | [upload or certificate manager reference] | [listener update] | [name, time] |
| [Java or .NET service] | [keystore or certificate store import] | [service restart] | [name, time] |
| [Appliance or device] | [vendor procedure reference] | [restart window] | [name, time] |
| [Clients that pin or trust it] | [pin set or trust bundle update] | [release or config push] | [name, time] |
6.3 Validate. Recovery is complete only when an external probe of every endpoint in 1.1 shows the new serial, the expected expiry and a complete chain, the dependent services in 1.1 report healthy, and the inventory record holds the new serial and expiry. Record the evidence.
6.4 Rollback. If the new certificate breaks clients (for example a chain or key type that older clients reject), restore the previous certificate only while it is still valid and not revoked, then fix forward. Never roll back to an expired or revoked certificate.
7. Communication and notification
- Internal: the Incident Manager updates [status channel] every [30] minutes for Sev 1, covering affected services, workaround and estimated fix time.
- Customers and partners: use [status page] for customer-facing outages. Tell partners whose integrations pin or trust this certificate before any planned change of chain, key or CA.
- CA: report suspected key compromise to the CA through its problem reporting route: [CA contact].
- Regulators: an expiry outage is normally an availability incident, not a data breach. Check [Organisation]'s incident classification for sector rules (for example DORA or NIS2 reporting thresholds for major incidents) and record the decision.
8. Post-incident review
8.1 Within [5] working days, the Certificate Owner leads a blameless review covering: time from first alert to detection and to recovery; why automation or monitoring did not prevent it; how many other certificates share the same weakness; and changes to this runbook. Track actions to closure in [tracker].
8.2 Record these measures each quarter for all runbooks: percentage of certificates renewed without human action; certificates expiring in production (target zero); renewals that succeeded but were not served on every endpoint; median days of remaining validity at renewal; and certificates in the served-certificate scan that are missing from the inventory.
Implementation guidance
- Start from the inventory: build the list of certificates and deployment targets from CLM, cloud certificate managers, CT logs and network scans before writing runbooks. Unknown certificates are the main cause of surprise expiry.
- Prioritise by lifetime and impact: write runbooks first for public TLS certificates on Tier 1 services, because they are the ones shortening to 200, 100 and then 47 days.
- Know what managed services will not renew: for example, AWS Certificate Manager documents that imported certificates, expired certificates and certificates issued through ACME are not eligible for its managed renewal, and that automatic renewal relies on DNS validation. Every managed service needs this check.
- Rehearse: run a renewal in a staging environment (Certbot provides
certbot renew --dry-run; Let's Encrypt applies lifetime changes to staging about a month before production), and run a tabletop of the incident path once a year. - Map to frameworks: this runbook supports NIST SP 800-53 SC-17 (Public Key Infrastructure Certificates), SC-12 (Cryptographic Key Establishment and Management), CM-8 (System Component Inventory) and IR-4 (Incident Handling), and NIST CSF 2.0 ID.AM-08 (systems managed throughout their life cycles) and RC.RP-01 (recovery plan executed).
- Link to policy: the rules this runbook applies (validity, key protection, approved CAs) belong in the Certificate Management Policy Template; automation design is covered in the ACME and Automated Certificate Management Guide.
Sources used for this template: CA/Browser Forum: TLS Baseline Requirements; CA/Browser Forum: Ballot SC-081v3; IETF: RFC 9773 ACME Renewal Information; Let's Encrypt: Decreasing Certificate Lifetimes to 45 Days; Certbot: User Guide; cert-manager: Certificate resource; AWS: ACM managed certificate renewal.
Related NHI Mgmt Group resources: Certificate Expiry Outages and 47-Day TLS Certificates Guide · CA Compromise and Mass Certificate Replacement Playbook · Certificate Revocation Guide · Certificate Management Policy Template