Join our Newsletter — 33% off our NHI Course
Home› Guides› Secrets manager day-2 operations runbook: recurring tasks, steps…
Template Secrets Management

Secrets manager day-2 operations runbook: recurring tasks, steps and evidence for platform owners

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 14 min read
On this page

A secrets manager that was hardened well at go-live drifts within months unless someone patches it, reviews who administers it, rotates its own root material, reads its audit trail and proves the backups restore. This template gives [Organisation] a day-2 runbook for the platform team that operates a secrets manager: the recurring tasks, who does them, the steps in order, the checks that confirm each step worked and the evidence to keep. It is vendor-neutral and applies to self-hosted clusters and managed cloud services alike, with notes where the two differ. It does not repeat hardening design, which is covered in Securing the secrets manager itself, or the per-secret rotation procedure, which lives in the Secrets Rotation Runbook Template. The scored quarterly review is the Secrets manager health check; this runbook is the work that keeps that review green.

How to use this template

  • One runbook per platform instance: fill in a copy for each cluster, tenant or managed service account that holds production secrets. Non-production instances can share a copy if they run the same version and configuration.
  • Replace the square brackets: text in [brackets] is a placeholder or guidance. Delete tasks that do not apply (for example, unseal key handling on a managed service where the provider holds the root of trust) and record why in section 1.
  • Cadence is a starting point: the intervals in section 3 suit a platform that serves production workloads. Shorten them after an incident or a failed task; lengthen them only with the platform owner's written approval.
  • Evidence is not optional: every task ends with a record. Auditors, the health check and the post-incident review all draw on these records.
  • Owner and review: [Secrets Platform Owner], reviewed every [12] months, after a major version upgrade and after any incident involving the platform.

1. Purpose and scope

1.1 This runbook defines the recurring operational tasks that keep [platform name and instance] secure and recoverable: patching and upgrades, administrative access reviews, rotation of the platform's own privileged credentials and root material, audit log review, and backup and restore testing.

1.2 In scope: server nodes or the managed service configuration, storage backend, agents and sidecars run by the platform team, the platform's authentication method configuration, its audit devices or log export, and recovery material.

1.3 Out of scope: rotating the application secrets the platform stores (use the Secrets Rotation Runbook Template), responding to a suspected compromise (use If your secrets manager is compromised) and architecture changes such as replication topology (use the [change process] and the Secrets Manager Resilience Guide).

1.4 Deployment model: [self-hosted / managed service / hybrid]. Tasks marked SH apply to self-hosted only; tasks marked MS apply to managed services only.

2. Roles and responsibilities

RoleResponsibilities in this runbookNamed holder
Platform ownerApproves the cadence, exceptions and maintenance windows; signs the quarterly evidence pack[Name]
Platform operator (primary and deputy)Runs patching, backup tests and log review; raises findings[Names]
Access reviewerIndependent of the operators; reviews administrator and policy-writer assignments[Name or role, for example IAM lead]
Recovery material custodiansHold unseal or recovery key shares or the equivalent; attend rekey and custody checks[Names, at least N for an M of N quorum]
Security operationsOwns detection rules on platform audit events and triages alerts[Team]
Application ownersConfirm their workloads recover after maintenance and restore tests[Distribution list]

2.1 No single person may both grant themselves platform administrator rights and approve that grant. Operators do not review their own access.

3. Operating calendar

3.1 The calendar below is the default. Record each run in the evidence log (section 9).

TaskCadenceSectionEvidence produced
Review security advisories for the server, agents and client librariesWeekly, and on vendor notification4Advisory log with decision per item
Apply patch or minor upgradeMonthly window, or within [7] days for a security fix rated critical4Change record, pre and post checks
Major version upgradePlanned, at least [6] months before the running version leaves support4Change record, rehearsal result
Triage high-risk audit eventsDaily (automated alerting, human triage)7Case records
Audit log pipeline and integrity checkWeekly7Pipeline check record
Administrative access reviewQuarterly, and on leaver or role change5Signed review with removals
Rotate platform service credentials (storage, replication, log export, integration accounts)Per credential lifetime in section 6, at most [90] days for static ones6Rotation record
Recovery material custody checkEvery [6] months6Custody register signed by each holder
Rekey or rotate the root of trustAnnually, and on custodian departure or suspected exposure6Ceremony record
Backup job verificationDaily automated, weekly human check8Backup report
Restore test into an isolated environmentQuarterly8Restore test record with timings
Break-glass access testEvery [6] months8Test record

4. Patching and upgrades

4.1 Watch: subscribe [operator distribution list] to the vendor's security advisories and release notes for the server, agents, plugins and client libraries. Record each advisory, whether it applies, and the decision with a target date. NIST SP 800-40 Rev. 4 treats patching as routine preventive maintenance; apply the same discipline here rather than waiting for an incident.

4.2 Plan the window: pick maintenance windows from past request volume and audit log data, as the OWASP Secrets Management Cheat Sheet recommends, because an outage can block the credentials needed to restore other services. Confirm that clients cache or retry, and tell application owners in advance.

4.3 Before you start (SH):

  1. Take and verify a fresh backup or storage snapshot (section 8.2), and record its identifier.
  2. Confirm recovery material custodians are reachable in case a node needs unsealing or re-initialising after restart.
  3. Read the upgrade notes for removed features, changed defaults and required upgrade order. Major versions often require upgrading standby or follower nodes before the active node.
  4. Run the upgrade in non-production first and run the post-checks in 4.5 there.

4.4 Apply (SH): upgrade one node at a time, waiting for each to rejoin and report healthy before the next. Upgrade the active node last so leadership moves once. For managed services (MS), confirm the provider's maintenance schedule, accept or defer optional upgrades within policy, and update agents and client libraries you run.

4.5 Post-checks: cluster health and replication status normal; every audit device or log export still enabled and delivering; a test client in each environment can authenticate and read a canary secret; dynamic credential engines can issue and revoke a test credential; error rates and latency back to baseline within [30] minutes.

4.6 Roll back: if post-checks fail and cannot be fixed within the window, stop. Where the vendor supports downgrade, revert node by node. Where storage format changed and downgrade is unsupported, restore the pre-upgrade snapshot to the previous version following section 8. Never restore an old snapshot over a cluster that has accepted writes without first exporting the changes made since, or recent secrets and leases will be lost.

5. Administrative access review

5.1 Export, at review time: every human and machine identity with administrator, policy-write, auth-method-write or audit-configuration rights; every root or equivalent all-powerful token in existence; and every identity that can change the platform's underlying infrastructure (cloud account roles, storage backend, hosts or orchestrator namespace).

5.2 For each entry the reviewer confirms: a named owner, a current business need, membership via a group rather than a direct grant, phishing-resistant MFA for humans, and use within the last [90] days. Unused or ownerless entries are removed, not marked "keep".

5.3 Root tokens: no standing root token should exist between ceremonies. If one is found, record who created it and why, revoke it, and raise a finding. Generate root access only for a defined task through the quorum process, then revoke it in the same session.

5.4 Infrastructure path: review the cloud or host roles that can read the storage backend, take snapshots, change key management settings for the seal key or alter log export. These paths bypass platform policy and are often missed. See PAM vs Secrets Manager for where privileged access tooling should broker these sessions.

5.5 Leavers: on the HR leaver or role change event, remove platform rights within [1] business day and check whether the person held recovery material (section 6.4).

5.6 Record removals, exceptions with expiry dates, and reviewer sign-off.

6. Rotating the platform's own credentials and root material

6.1 This section covers credentials the platform uses or depends on, not the secrets it stores. Keep a register:

CredentialTypical formRotation approachLifetime target
Storage backend accessDatabase account, object storage keyPrefer platform or workload identity; otherwise dual credential swap[90] days or identity-based
Log export or SIEM forwardingAPI token, client certificateIssue new, confirm delivery, revoke old[90] days
Replication and cluster TLSNode certificatesAutomated renewal with alerting at [30] days to expiryPer internal PKI policy
Seal or wrapping key in an external KMS or HSM (SH)Key reference plus access credentialRotate key version per KMS policy; rotate access credential separatelyPer key management policy
Unseal or recovery key shares (SH)M of N shares held by custodiansRekey ceremonyAnnually and on custodian change
Break-glass credentialSealed account held in a separate systemRotate after every use and every testAfter each use

6.2 Service credentials: issue the new credential, configure the platform to use it, confirm the dependent function works (storage writes, log delivery, replication), then revoke the old one. Follow the steps and rollback in the Secrets Rotation Runbook Template.

6.3 Where these credentials live: the OWASP cheat sheet advises keeping the root and management credentials of a secrets solution in a secondary system. Do not store the platform's own recovery material or break-glass credential inside the platform it protects. The Secrets Zero Problem Guide covers the bootstrap trade-offs.

6.4 Rekey ceremony (SH):

  1. Schedule with at least the quorum of current custodians plus the new custodians, and a witness from [security or audit].
  2. Generate a short-lived root or use the documented rekey operation, whichever the platform requires.
  3. Issue new shares, each encrypted to the receiving custodian (for example with their personal public key), and never displayed together on one screen.
  4. Confirm the new shares work by completing an unseal or recovery operation in non-production or a verification step the platform provides, before the old shares are destroyed.
  5. Destroy old shares, revoke any root token generated, and update the custody register.

Run the ceremony with the Cryptographic Key Ceremony Template.

6.5 Custody check: each custodian confirms they still hold their share in the agreed storage, without revealing it. If available holders fall to the quorum number, rekey now.

7. Audit log review

7.1 Pipeline check (weekly): confirm every node or region delivered audit events in the last 24 hours, the event count is within the normal range, and the destination is write-protected from platform operators. NIST SP 800-53 AU-9 asks for audit information to be protected from unauthorised modification and deletion. Check clock synchronisation: OWASP notes that timestamps must be accurate and clock changes monitored.

7.2 Plaintext check: sample [20] events and confirm secret values and tokens are hashed or absent. Plaintext secrets in logs turn the SIEM into a second secret store.

7.3 Events to alert on and triage within [24] hours:

EventWhy it mattersFirst triage question
Root token generated or usedBypasses all policyIs there an approved ceremony or change record?
Audit device disabled, removed or failingBlinds detectionWas it a planned change, and is a second sink still receiving?
Policy or role binding created or widenedCommon path to bulk accessDoes the change match an approved request?
Authentication method added or reconfiguredCan let a new identity source inWho made it, and which identities can now log in?
Bulk reads or list operations across many pathsPossible harvestingIs the identity a known sync or backup job?
Repeated authentication or permission errors from one clientProbing or a broken deploymentDid the client change recently?
Snapshot, export or replication target changeCopies all secrets elsewhereIs the destination approved?

7.4 Monthly review: beyond alerts, read a summary of administrative actions for the month and confirm each maps to a change or request. This catches slow drift that no single alert flags. For detection rule design across log sources, see the CIS Control 8 Audit Log Management Guide.

7.5 Escalate any event that cannot be explained to security operations under the compromise playbook.

8. Backup and restore testing

8.1 Backups: the OWASP cheat sheet recommends automated periodic backups, regular restore tests, encryption of backups, reduced access rights to them and monitoring of their location. Record backup frequency, location, encryption and who can read them: [values].

8.2 Daily verification: the job completed, the artefact size is within the normal range, and the artefact is stored in a location the platform operators cannot delete from on their own (for example an immutable or separately administered store).

8.3 Quarterly restore test:

  1. Restore the latest backup into an isolated environment with no network path to production clients.
  2. Unseal or initialise it with the real recovery process and real custodians (SH), or the provider's restore feature (MS). A test that skips the custodians does not prove recovery.
  3. Read [5] canary secrets and compare their versions with production; confirm policies and auth methods restored.
  4. Record time to restore and data loss window against the agreed recovery time and recovery point objectives.
  5. Destroy the restored copy and any tokens issued during the test, and record that you did.

8.4 Break-glass test: retrieve the break-glass credential from its separate store, confirm it works, confirm the alert fired, then rotate it. See the Break Glass Access Procedure Template.

8.5 A failed restore is a priority finding: fix and retest within [30] days, and report it to the platform owner.

9. Evidence log and exceptions

9.1 Keep one evidence log per instance with: date, task, operator, result (pass, fail, partial), ticket or change reference, and link to artefacts. Retain for [the audit retention period, for example 3 years].

9.2 A missed task is an exception: record the reason, the risk and the new date, approved by the platform owner. Two consecutive misses of the same task go to [risk committee].

9.3 Feed the quarterly evidence pack into the Secrets manager health check, which scores the outcomes. This runbook does not define new metrics; track only task completion on time and failures found per task, so the two documents do not drift apart.

Implementation guidance

  1. Start with the register: before scheduling anything, list instances, administrators, recovery material holders, platform service credentials and backup locations. Most first runs find a forgotten root token or an ex-employee holding a key share.
  2. Automate the checks, not the judgement: pipeline health, backup completion, certificate expiry and advisory feeds should alert automatically. Access reviews and restore tests need people.
  3. Rehearse once before the first real run: do a restore test and a rekey in non-production so custodians learn the process when nothing is at stake.
  4. Small teams: combine roles but keep the reviewer independent of the operators, even if the reviewer is from another team. Managed services remove the unseal and node patching tasks but not the access review, log review or restore test.
  5. Large estates: run one calendar for all instances, staggered so that not every cluster is patched in the same window, and report completion per instance.
  6. Map to frameworks: the tasks support NIST SP 800-53 SI-2 Flaw Remediation, AC-2 Account Management, AC-6 Least Privilege, IA-5 Authenticator Management, SC-12 Cryptographic Key Establishment and Management, AU-6 Audit Record Review, Analysis, and Reporting, AU-9 Protection of Audit Information, CP-9 System Backup and CP-4 Contingency Plan Testing; and NIST CSF 2.0 PR.PS-02, PR.AA-05, PR.PS-04, DE.CM-09, PR.DS-11 and RC.RP-03. If you run a secrets operator in Kubernetes, add its controller credentials to section 6; see the External Secrets Operator security guide.

Sources used for this template: OWASP: Secrets Management Cheat Sheet; NIST: SP 800-40 Rev. 4 Guide to Enterprise Patch Management Planning; NIST: SP 800-53 Rev. 5; NIST: Cybersecurity Framework 2.0; NIST: SP 800-57 Part 1 Rev. 5 Recommendation for Key Management; NIST: SP 800-92 Rev. 1 (Initial Public Draft) Cybersecurity Log Management Planning Guide.

Related NHI Mgmt Group resources: Securing the secrets manager itself · Secrets manager health check · Secrets Rotation Runbook Template · If your secrets manager is compromised · Hardware security module (HSM) day-2 operations runbook

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.