A secrets manager only reduces risk once applications actually fetch their secrets from it, and many rollouts end with a well-built platform that holds copies of secrets still sitting in code, pipelines and configuration files. This playbook covers deploying a secrets manager as a product class: the design decisions to make before go-live, how to build the platform securely, how to onboard applications in waves, how to test and roll back, and what to measure. It is vendor-neutral and based on the OWASP Secrets Management Cheat Sheet, NIST SP 800-53 and SP 800-57, and CIS Controls v8.1. It is written for platform engineers, DevOps leads and security architects. The programme-level roadmap and principles are in the Secrets Management Guide; product selection is in the Secrets Management Buyer's Guide.
Key takeaways
- Design the root of trust first. OWASP advises keeping the root or management credentials of the secrets store somewhere other than the store itself; decide where the master key, unseal or recovery material and the first admin credential live before installing anything.
- Machine authentication decides the security of the whole platform. Use platform identity (workload identity, OIDC federation, cloud instance identity) so applications do not need a static secret to fetch their secrets.
- Onboard by consumption pattern, not by team. Group applications by how they will receive secrets (runtime API, sidecar or agent, injected file, pipeline) and build one paved road per pattern.
- Migration is not finished until the old copy is gone. Each wave must rotate the secret after moving it and remove the copy from code, pipelines and configuration.
- The platform is now critical infrastructure. OWASP stresses high availability because applications depend on the service at start-up; test backup, restore and break-glass before production traffic.
Phase 0: platform design decisions
These decisions shape every later phase. Changing them after applications are onboarded means repeating the onboarding.
| Decision | Options | Security built in |
|---|---|---|
| Deployment model | Cloud-native service per cloud, self-hosted cluster, or a cloud-agnostic platform across environments | OWASP notes a single solution across clouds standardises policy, rotation and audit; native services reduce operational load. Either way, standardise lifecycle, authentication, authorisation and accounting |
| Tenancy and separation | Separate instances, namespaces, projects or vaults per environment and application | OWASP advises separating production and development secrets, ideally into different managers |
| Root of trust | Software key, hardware security module or cloud key service for the master or key-encryption key | Envelope encryption with the key held outside the data store (OWASP sections 4.2 and 7.3); key handling under NIST SP 800-57 |
| Machine authentication | Workload identity, OIDC federation, cloud instance identity, mutual TLS, or static tokens | Platform identity by default; static tokens only by exception, scoped and short-lived. See the Secrets Zero Problem Guide |
| Human authentication | Organisation identity provider with MFA | Phishing-resistant MFA for administrators (CIS safeguard 6.5); separate admin accounts (5.4) |
| Naming and metadata | Path or tag convention for environment, application, owner, secret type, rotation schedule | OWASP section 2.11: record history, owner, purpose, type and rotation schedule. Policies can then be written against paths and tags |
| Availability | Replicas, regions, caching and quotas | Plan for API limits (OWASP section 4.4) and failover. See the Secrets Manager Resilience Guide |
Phase 1: build and harden the platform
- Install with infrastructure as code: configuration, policies and authentication methods live in version control with review, so any change to who can read what is a reviewed change (NIST SP 800-53 CM-3).
- Protect the initial administrator credential: use it once to configure identity-provider login and named admin roles, then revoke it or seal it away under split knowledge. Keep recovery material in a secondary system, as OWASP advises for break-glass credentials.
- Define admin roles narrowly: platform operators (configuration, no secret read), policy authors, auditors (read-only logs) and application owners (their own paths). OWASP section 2.3: engineers should not be able to read every secret.
- Turn on audit logging before the first secret: log secret reads, writes, policy changes, authentication failures and administrative actions to a store the platform administrators cannot alter (OWASP section 2.6; AU-9).
- Restrict network reach: TLS everywhere (OWASP section 2.8); expose the API only to networks that host consumers and admin devices.
- Prove recovery: automate encrypted backups, restore to an isolated environment and run break-glass before onboarding production (OWASP section 2.9).
Hardening detail for the platform itself belongs in a health check; the planned Secrets Management Checklist can be used as the go-live gate.
Phase 2: discovery and onboarding backlog
Before onboarding, find where secrets live today. Scan repositories, CI/CD variables, container images, configuration management, infrastructure-as-code state and collaboration tools. For each secret record: owner, consuming application, environment, what it unlocks, and how it is delivered today. Rank the backlog by impact and exposure: production secrets with broad access, secrets already found in code, and secrets with no owner come first. Secrets found in code are treated as exposed and rotated, not merely moved; the Toyota T-Connect Key Exposure 2022 shows a key that sat in a public repository for almost five years before it was changed.
Phase 3: pilot one paved road per pattern
Pick one application per consumption pattern and build the reusable integration, documentation and policy template for it. OWASP section 6.5 notes that adoption depends on usability: SDKs, CLI, plugins for CI/CD and infrastructure-as-code, self-service for low-risk secrets and clear errors.
| Pattern | How the application gets the secret | Choose when | Watch for |
|---|---|---|---|
| Runtime API or SDK | The application authenticates with its platform identity and fetches at start-up or on demand | New and actively maintained services | Caching and retry so a platform outage does not cascade |
| Sidecar or agent | A helper authenticates, writes secrets to an in-memory volume and refreshes them (OWASP section 2.4.1) | Containerised workloads, applications you cannot change much | Shared volumes across deployments (OWASP section 5.3) |
| Orchestrator-injected file | The orchestrator mounts the secret as a file at runtime | Platforms with a native secret object synchronised from the manager | Encrypting the orchestrator's own store and limiting who can read it |
| Pipeline retrieval | The pipeline authenticates with OIDC federation and fetches short-lived credentials (OWASP section 3.2.2) | Deployment-time credentials | Secrets echoed to build logs; long-lived pipeline variables |
| Not touched by the pipeline | The pipeline deploys; the running application fetches its own secrets (OWASP section 3.2.3) | Preferred where possible | The deployed identity must be scoped to its own paths |
OWASP discourages environment variables because they leak through logs, dumps and child processes. Where they are unavoidable, see the Environment Variables and .env Files.
Phase 4: onboarding waves
Our recommended order, after the pilot:
- Wave 1, exposed and high-impact secrets: cloud keys, database credentials and signing keys for production, and any secret found in code during discovery.
- Wave 2, CI/CD: replace long-lived cloud keys in pipelines with OIDC federation, then move remaining pipeline secrets into the manager.
- Wave 3, application estate by pattern: use the paved roads; each team self-serves with the policy template.
- Wave 4, dynamic and short-lived: move databases and cloud access to dynamic credentials issued per session with short leases (OWASP section 6.2). See the Dynamic Secrets and Ephemeral Credentials Guide.
- Wave 5, long tail: third-party API keys, webhook secrets, legacy applications, with dated exceptions for anything that cannot move.
Per-application cutover, testing and rollback
- Store the current secret in the manager with owner and metadata, and grant the application's identity read access to its path only.
- Deploy the integration in a non-production environment and confirm the application reads from the manager.
- Rotate the secret: issue a new value through the manager, using the create, set, test, finish sequence OWASP describes, and confirm the application picks it up.
- Remove the old copy from code, pipeline variables and configuration; revoke the old value.
- Rollback: revert the application to the previous release that reads from the manager with the prior version of the secret, never to a hard-coded value. Version history in the manager makes this possible.
Moving between two managers is a different problem; see the Secrets manager migration playbook.
Small versus large organisations
- Small teams: a cloud-native service in one cloud, workload identity for compute, OIDC for the pipeline and a single path convention will cover most needs. Skip self-hosting unless a requirement forces it.
- Large or multi-cloud estates: a central platform team runs the manager or federates several, publishes paved roads and policy templates, and measures adoption by business unit. Delegate path ownership to application teams; keep policy review central.
- Regulated environments: OWASP notes hardware security modules protect keys rather than application secrets; use one for the root key where policy requires it.
Detection and measurement
OWASP section 6.4 advises tracking who or what accessed each secret, from where and how, and treating anomalies such as a pipeline credential used from an unexpected address as likely compromise. Build alerts for: reads by an identity outside its usual set of paths, bulk reads, policy changes that broaden access, authentication failures from workload identities, root or recovery credential use, and audit device failures. See Secrets Access Logging and Detection.
| Metric | How to measure | Target (our recommendation) |
|---|---|---|
| Adoption | Production applications reading secrets from the manager, divided by production applications | Rising each quarter, tracked per team |
| Secrets outside the manager | Open scanner findings for live secrets in code, pipelines and images | Falling; new findings closed within the policy window |
| Static versus dynamic | Share of credentials that are dynamic or replaced by workload identity | Rising |
| Rotation compliance | Static secrets rotated within their scheduled period | 95% or more |
| Over-broad policies | Policies granting wildcard read across applications or environments | Zero outside break-glass |
| Recovery readiness | Days since last successful restore test and break-glass test | Within the period set in the resilience plan |
Framework mapping
| Rollout element | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | CIS Controls v8.1 |
|---|---|---|---|
| Remove secrets from code and configuration | IA-5(7) No Embedded Unencrypted Static Authenticators | PR.AA-01 | 5.5 (service account inventory) |
| Authenticator lifecycle and rotation | IA-5 Authenticator Management | PR.AA-01 | 5.2 |
| Least-privilege policies | AC-6(5) Privileged Accounts, AC-3 Access Enforcement | PR.AA-05 | 6.8 |
| Key management and encryption at rest | SC-12 Cryptographic Key Establishment and Management, SC-28 Protection of Information at Rest | PR.DS-01 | 3.11 |
| Audit and detection | AU-9 Protection of Audit Information, SI-4 System Monitoring | PR.PS-04, DE.CM-09 | 8.2 |
| Resilience | CP-9 System Backup, CP-10 System Recovery and Reconstitution | PR.IR-03 | 11.2 |
The ATT&CK technique the rollout most directly reduces is T1552.001 Credentials In Files.
Common failure patterns
- Moving without rotating: the secret is copied into the manager but the old value still works and still sits in history.
- A static token to fetch every secret: applications authenticate with a long-lived token stored in an environment variable, recreating the original problem one level up.
- Wildcard policies to speed onboarding: broad read access granted "temporarily" becomes permanent.
- No owner metadata: rotation and incident response stall because nobody knows what breaks if a secret changes.
- Single point of failure: applications that fetch on every request fail together when the platform or its quota is exhausted.
- Audit logging added later: early access to production secrets has no trail when an incident needs one.
Practitioner checklist
- Decide deployment model, tenancy and root of trust before installation.
- Seal the initial admin credential and keep recovery material in a separate system.
- Enable tamper-resistant audit logging before storing the first secret.
- Make platform identity the default authentication method for workloads and pipelines.
- Publish one paved road and policy template per consumption pattern.
- Rotate every secret during cutover and remove the old copy.
- Test restore and break-glass before onboarding production.
- Report adoption, secrets outside the manager and rotation compliance monthly.
Standards and references
- OWASP: Secrets Management Cheat Sheet
- NIST: SP 800-57 Part 1 Rev. 5 Recommendation for Key Management
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- CIS: Controls v8.1 Assessment Specification, Control 5 Account Management
- CIS: Controls v8.1 Assessment Specification, Control 6 Access Control Management
- CIS: Controls v8.1 Assessment Specification, Control 3 Data Protection
- CIS: Controls v8.1 Assessment Specification, Control 8 Audit Log Management
- CIS: Controls v8.1 Assessment Specification, Control 11 Data Recovery
- MITRE ATT&CK: Credentials In Files
Related NHI Mgmt Group resources: Secrets Management Maturity Model · Secrets Management Policy Template · Secrets Rotation Runbook Template · Privileged access management (PAM) vault rollout playbook