A secrets manager that was hardened well at go-live drifts within months unless someone patches it, reviews who administers it, rotates its own root material, reads its audit trail and proves the backups restore. This template gives [Organisation] a day-2 runbook for the platform team that operates a secrets manager: the recurring tasks, who does them, the steps in order, the checks that confirm each step worked and the evidence to keep. It is vendor-neutral and applies to self-hosted clusters and managed cloud services alike, with notes where the two differ. It does not repeat hardening design, which is covered in Securing the secrets manager itself, or the per-secret rotation procedure, which lives in the Secrets Rotation Runbook Template. The scored quarterly review is the Secrets manager health check; this runbook is the work that keeps that review green.
How to use this template
- One runbook per platform instance: fill in a copy for each cluster, tenant or managed service account that holds production secrets. Non-production instances can share a copy if they run the same version and configuration.
- Replace the square brackets: text in [brackets] is a placeholder or guidance. Delete tasks that do not apply (for example, unseal key handling on a managed service where the provider holds the root of trust) and record why in section 1.
- Cadence is a starting point: the intervals in section 3 suit a platform that serves production workloads. Shorten them after an incident or a failed task; lengthen them only with the platform owner's written approval.
- Evidence is not optional: every task ends with a record. Auditors, the health check and the post-incident review all draw on these records.
- Owner and review: [Secrets Platform Owner], reviewed every [12] months, after a major version upgrade and after any incident involving the platform.
1. Purpose and scope
1.1 This runbook defines the recurring operational tasks that keep [platform name and instance] secure and recoverable: patching and upgrades, administrative access reviews, rotation of the platform's own privileged credentials and root material, audit log review, and backup and restore testing.
1.2 In scope: server nodes or the managed service configuration, storage backend, agents and sidecars run by the platform team, the platform's authentication method configuration, its audit devices or log export, and recovery material.
1.3 Out of scope: rotating the application secrets the platform stores (use the Secrets Rotation Runbook Template), responding to a suspected compromise (use If your secrets manager is compromised) and architecture changes such as replication topology (use the [change process] and the Secrets Manager Resilience Guide).
1.4 Deployment model: [self-hosted / managed service / hybrid]. Tasks marked SH apply to self-hosted only; tasks marked MS apply to managed services only.
2. Roles and responsibilities
| Role | Responsibilities in this runbook | Named holder |
|---|---|---|
| Platform owner | Approves the cadence, exceptions and maintenance windows; signs the quarterly evidence pack | [Name] |
| Platform operator (primary and deputy) | Runs patching, backup tests and log review; raises findings | [Names] |
| Access reviewer | Independent of the operators; reviews administrator and policy-writer assignments | [Name or role, for example IAM lead] |
| Recovery material custodians | Hold unseal or recovery key shares or the equivalent; attend rekey and custody checks | [Names, at least N for an M of N quorum] |
| Security operations | Owns detection rules on platform audit events and triages alerts | [Team] |
| Application owners | Confirm their workloads recover after maintenance and restore tests | [Distribution list] |
2.1 No single person may both grant themselves platform administrator rights and approve that grant. Operators do not review their own access.
3. Operating calendar
3.1 The calendar below is the default. Record each run in the evidence log (section 9).
| Task | Cadence | Section | Evidence produced |
|---|---|---|---|
| Review security advisories for the server, agents and client libraries | Weekly, and on vendor notification | 4 | Advisory log with decision per item |
| Apply patch or minor upgrade | Monthly window, or within [7] days for a security fix rated critical | 4 | Change record, pre and post checks |
| Major version upgrade | Planned, at least [6] months before the running version leaves support | 4 | Change record, rehearsal result |
| Triage high-risk audit events | Daily (automated alerting, human triage) | 7 | Case records |
| Audit log pipeline and integrity check | Weekly | 7 | Pipeline check record |
| Administrative access review | Quarterly, and on leaver or role change | 5 | Signed review with removals |
| Rotate platform service credentials (storage, replication, log export, integration accounts) | Per credential lifetime in section 6, at most [90] days for static ones | 6 | Rotation record |
| Recovery material custody check | Every [6] months | 6 | Custody register signed by each holder |
| Rekey or rotate the root of trust | Annually, and on custodian departure or suspected exposure | 6 | Ceremony record |
| Backup job verification | Daily automated, weekly human check | 8 | Backup report |
| Restore test into an isolated environment | Quarterly | 8 | Restore test record with timings |
| Break-glass access test | Every [6] months | 8 | Test record |
4. Patching and upgrades
4.1 Watch: subscribe [operator distribution list] to the vendor's security advisories and release notes for the server, agents, plugins and client libraries. Record each advisory, whether it applies, and the decision with a target date. NIST SP 800-40 Rev. 4 treats patching as routine preventive maintenance; apply the same discipline here rather than waiting for an incident.
4.2 Plan the window: pick maintenance windows from past request volume and audit log data, as the OWASP Secrets Management Cheat Sheet recommends, because an outage can block the credentials needed to restore other services. Confirm that clients cache or retry, and tell application owners in advance.
4.3 Before you start (SH):
- Take and verify a fresh backup or storage snapshot (section 8.2), and record its identifier.
- Confirm recovery material custodians are reachable in case a node needs unsealing or re-initialising after restart.
- Read the upgrade notes for removed features, changed defaults and required upgrade order. Major versions often require upgrading standby or follower nodes before the active node.
- Run the upgrade in non-production first and run the post-checks in 4.5 there.
4.4 Apply (SH): upgrade one node at a time, waiting for each to rejoin and report healthy before the next. Upgrade the active node last so leadership moves once. For managed services (MS), confirm the provider's maintenance schedule, accept or defer optional upgrades within policy, and update agents and client libraries you run.
4.5 Post-checks: cluster health and replication status normal; every audit device or log export still enabled and delivering; a test client in each environment can authenticate and read a canary secret; dynamic credential engines can issue and revoke a test credential; error rates and latency back to baseline within [30] minutes.
4.6 Roll back: if post-checks fail and cannot be fixed within the window, stop. Where the vendor supports downgrade, revert node by node. Where storage format changed and downgrade is unsupported, restore the pre-upgrade snapshot to the previous version following section 8. Never restore an old snapshot over a cluster that has accepted writes without first exporting the changes made since, or recent secrets and leases will be lost.
5. Administrative access review
5.1 Export, at review time: every human and machine identity with administrator, policy-write, auth-method-write or audit-configuration rights; every root or equivalent all-powerful token in existence; and every identity that can change the platform's underlying infrastructure (cloud account roles, storage backend, hosts or orchestrator namespace).
5.2 For each entry the reviewer confirms: a named owner, a current business need, membership via a group rather than a direct grant, phishing-resistant MFA for humans, and use within the last [90] days. Unused or ownerless entries are removed, not marked "keep".
5.3 Root tokens: no standing root token should exist between ceremonies. If one is found, record who created it and why, revoke it, and raise a finding. Generate root access only for a defined task through the quorum process, then revoke it in the same session.
5.4 Infrastructure path: review the cloud or host roles that can read the storage backend, take snapshots, change key management settings for the seal key or alter log export. These paths bypass platform policy and are often missed. See PAM vs Secrets Manager for where privileged access tooling should broker these sessions.
5.5 Leavers: on the HR leaver or role change event, remove platform rights within [1] business day and check whether the person held recovery material (section 6.4).
5.6 Record removals, exceptions with expiry dates, and reviewer sign-off.
6. Rotating the platform's own credentials and root material
6.1 This section covers credentials the platform uses or depends on, not the secrets it stores. Keep a register:
| Credential | Typical form | Rotation approach | Lifetime target |
|---|---|---|---|
| Storage backend access | Database account, object storage key | Prefer platform or workload identity; otherwise dual credential swap | [90] days or identity-based |
| Log export or SIEM forwarding | API token, client certificate | Issue new, confirm delivery, revoke old | [90] days |
| Replication and cluster TLS | Node certificates | Automated renewal with alerting at [30] days to expiry | Per internal PKI policy |
| Seal or wrapping key in an external KMS or HSM (SH) | Key reference plus access credential | Rotate key version per KMS policy; rotate access credential separately | Per key management policy |
| Unseal or recovery key shares (SH) | M of N shares held by custodians | Rekey ceremony | Annually and on custodian change |
| Break-glass credential | Sealed account held in a separate system | Rotate after every use and every test | After each use |
6.2 Service credentials: issue the new credential, configure the platform to use it, confirm the dependent function works (storage writes, log delivery, replication), then revoke the old one. Follow the steps and rollback in the Secrets Rotation Runbook Template.
6.3 Where these credentials live: the OWASP cheat sheet advises keeping the root and management credentials of a secrets solution in a secondary system. Do not store the platform's own recovery material or break-glass credential inside the platform it protects. The Secrets Zero Problem Guide covers the bootstrap trade-offs.
6.4 Rekey ceremony (SH):
- Schedule with at least the quorum of current custodians plus the new custodians, and a witness from [security or audit].
- Generate a short-lived root or use the documented rekey operation, whichever the platform requires.
- Issue new shares, each encrypted to the receiving custodian (for example with their personal public key), and never displayed together on one screen.
- Confirm the new shares work by completing an unseal or recovery operation in non-production or a verification step the platform provides, before the old shares are destroyed.
- Destroy old shares, revoke any root token generated, and update the custody register.
Run the ceremony with the Cryptographic Key Ceremony Template.
6.5 Custody check: each custodian confirms they still hold their share in the agreed storage, without revealing it. If available holders fall to the quorum number, rekey now.
7. Audit log review
7.1 Pipeline check (weekly): confirm every node or region delivered audit events in the last 24 hours, the event count is within the normal range, and the destination is write-protected from platform operators. NIST SP 800-53 AU-9 asks for audit information to be protected from unauthorised modification and deletion. Check clock synchronisation: OWASP notes that timestamps must be accurate and clock changes monitored.
7.2 Plaintext check: sample [20] events and confirm secret values and tokens are hashed or absent. Plaintext secrets in logs turn the SIEM into a second secret store.
7.3 Events to alert on and triage within [24] hours:
| Event | Why it matters | First triage question |
|---|---|---|
| Root token generated or used | Bypasses all policy | Is there an approved ceremony or change record? |
| Audit device disabled, removed or failing | Blinds detection | Was it a planned change, and is a second sink still receiving? |
| Policy or role binding created or widened | Common path to bulk access | Does the change match an approved request? |
| Authentication method added or reconfigured | Can let a new identity source in | Who made it, and which identities can now log in? |
| Bulk reads or list operations across many paths | Possible harvesting | Is the identity a known sync or backup job? |
| Repeated authentication or permission errors from one client | Probing or a broken deployment | Did the client change recently? |
| Snapshot, export or replication target change | Copies all secrets elsewhere | Is the destination approved? |
7.4 Monthly review: beyond alerts, read a summary of administrative actions for the month and confirm each maps to a change or request. This catches slow drift that no single alert flags. For detection rule design across log sources, see the CIS Control 8 Audit Log Management Guide.
7.5 Escalate any event that cannot be explained to security operations under the compromise playbook.
8. Backup and restore testing
8.1 Backups: the OWASP cheat sheet recommends automated periodic backups, regular restore tests, encryption of backups, reduced access rights to them and monitoring of their location. Record backup frequency, location, encryption and who can read them: [values].
8.2 Daily verification: the job completed, the artefact size is within the normal range, and the artefact is stored in a location the platform operators cannot delete from on their own (for example an immutable or separately administered store).
8.3 Quarterly restore test:
- Restore the latest backup into an isolated environment with no network path to production clients.
- Unseal or initialise it with the real recovery process and real custodians (SH), or the provider's restore feature (MS). A test that skips the custodians does not prove recovery.
- Read [5] canary secrets and compare their versions with production; confirm policies and auth methods restored.
- Record time to restore and data loss window against the agreed recovery time and recovery point objectives.
- Destroy the restored copy and any tokens issued during the test, and record that you did.
8.4 Break-glass test: retrieve the break-glass credential from its separate store, confirm it works, confirm the alert fired, then rotate it. See the Break Glass Access Procedure Template.
8.5 A failed restore is a priority finding: fix and retest within [30] days, and report it to the platform owner.
9. Evidence log and exceptions
9.1 Keep one evidence log per instance with: date, task, operator, result (pass, fail, partial), ticket or change reference, and link to artefacts. Retain for [the audit retention period, for example 3 years].
9.2 A missed task is an exception: record the reason, the risk and the new date, approved by the platform owner. Two consecutive misses of the same task go to [risk committee].
9.3 Feed the quarterly evidence pack into the Secrets manager health check, which scores the outcomes. This runbook does not define new metrics; track only task completion on time and failures found per task, so the two documents do not drift apart.
Implementation guidance
- Start with the register: before scheduling anything, list instances, administrators, recovery material holders, platform service credentials and backup locations. Most first runs find a forgotten root token or an ex-employee holding a key share.
- Automate the checks, not the judgement: pipeline health, backup completion, certificate expiry and advisory feeds should alert automatically. Access reviews and restore tests need people.
- Rehearse once before the first real run: do a restore test and a rekey in non-production so custodians learn the process when nothing is at stake.
- Small teams: combine roles but keep the reviewer independent of the operators, even if the reviewer is from another team. Managed services remove the unseal and node patching tasks but not the access review, log review or restore test.
- Large estates: run one calendar for all instances, staggered so that not every cluster is patched in the same window, and report completion per instance.
- Map to frameworks: the tasks support NIST SP 800-53 SI-2 Flaw Remediation, AC-2 Account Management, AC-6 Least Privilege, IA-5 Authenticator Management, SC-12 Cryptographic Key Establishment and Management, AU-6 Audit Record Review, Analysis, and Reporting, AU-9 Protection of Audit Information, CP-9 System Backup and CP-4 Contingency Plan Testing; and NIST CSF 2.0 PR.PS-02, PR.AA-05, PR.PS-04, DE.CM-09, PR.DS-11 and RC.RP-03. If you run a secrets operator in Kubernetes, add its controller credentials to section 6; see the External Secrets Operator security guide.
Sources used for this template: OWASP: Secrets Management Cheat Sheet; NIST: SP 800-40 Rev. 4 Guide to Enterprise Patch Management Planning; NIST: SP 800-53 Rev. 5; NIST: Cybersecurity Framework 2.0; NIST: SP 800-57 Part 1 Rev. 5 Recommendation for Key Management; NIST: SP 800-92 Rev. 1 (Initial Public Draft) Cybersecurity Log Management Planning Guide.
Related NHI Mgmt Group resources: Securing the secrets manager itself · Secrets manager health check · Secrets Rotation Runbook Template · If your secrets manager is compromised · Hardware security module (HSM) day-2 operations runbook