How to migrate Kubernetes secrets to an external Secrets Manager

Native Kubernetes Secrets usually start life in a YAML file, a Helm values file or a CI variable, so the same password ends up in Git history, in etcd and on every laptop that ran the deploy. Moving the source of truth to an external secrets manager (AWS Secrets Manager, Azure Key Vault, Google Secret Manager, HashiCorp Vault or similar) gives one place to control access, audit reads and rotate values, while workloads keep reading a Secret or a mounted file. This page is the migration procedure: inventory what you have, choose between the External Secrets Operator (ESO) and the Secrets Store CSI Driver, seed the external store without copying values by hand, cut workloads over, roll back if needed, and retire the old copies. It is for platform engineers and Kubernetes administrators. Hardening the operator itself is covered in the External Secrets Operator security guide, and Secret security inside the cluster in the Kubernetes Secrets Security Guide.

This is an independent guide from NHI Mgmt Group, based on the public documentation of Kubernetes, the External Secrets Operator project and the Secrets Store CSI Driver project as of November 2026. It is not sponsored or reviewed by any of these projects or by any secrets manager vendor. Field names and defaults change between releases, so confirm against the documentation for the version you run.

Key takeaways

• Migration is not finished until the old value is rotated. Every value that lived in Git, Helm values or etcd without encryption must be treated as exposed. Move it, then rotate it in the external store and let the sync deliver the new value.
• Seed the store from the cluster, not from laptops. ESO's PushSecret can copy an existing Secret's keys into the external store, so nobody has to decode and paste values by hand.
• Use a safe creation policy during cut-over. With creationPolicy: Orphan the Secret survives if you delete the ExternalSecret, which makes rollback a one-line change. Switch to Owner once stable.
• Environment variables do not refresh. Kubernetes updates mounted Secret volumes eventually, but a Pod reading a Secret through an environment variable or a subPath mount only sees a new value after restart. Plan restarts with each rotation.
• The cluster still holds a credential. ESO or the CSI provider needs its own identity to read the store. Use workload identity federation, not a static access key in another Secret.

When you need this

• Secrets are committed to Git or Helm values. Even base64-encoded, they are plaintext to anyone who can read the repository, including its history. Exposed cluster secrets have unlocked far more than the cluster: in the SAP exposed Kubernetes secrets incident, a key with access to an artifact repository was found in public.
• Rotation means editing YAML in several places. An external store gives one value to rotate and lets the sync push it to every namespace and cluster that uses it.
• You need to know who read a secret. Kubernetes audit logs record API access, but cloud secrets managers and Vault add per-secret access logging and policy outside the cluster.
• Several clusters share credentials. Copying the same Secret into each cluster multiplies exposure; one store with per-cluster read policies does not.

Before you start

Choosing the delivery mechanism

Most estates migrate with ESO first, because it changes nothing in the workload, then move the most sensitive applications to CSI volumes or direct reads later. Git-based alternatives such as SOPS or Sealed Secrets solve the Git problem but not central rotation and access audit; see SOPS and Sealed Secrets for that comparison.

Step-by-step

Steps 1 and 2 are read-only. Steps 3 to 5 add the external path alongside the existing Secret. Step 6 changes ownership. Steps 7 and 8 retire the old copies and are the destructive part.

1. Inventory Secrets without printing values. kubectl get and kubectl describe show names, types and key names but not the contents:

   kubectl get secrets --all-namespaces
   kubectl describe secret <name> -n <namespace>

   Export name, namespace, type, key names, labels, annotations and ownerReferences to a dated file. Do not use -o yaml or -o jsonpath='{.data}' for the inventory: that decodes to the values, and the file becomes a secret itself.
2. Classify each Secret. Not every Secret should move.

   Secret	Action
   Opaque, kubernetes.io/basic-auth, kubernetes.io/ssh-auth created from manifests or CI	Migrate: these are the application credentials the project exists to move
   kubernetes.io/dockerconfigjson image pull Secrets	Migrate with an ExternalSecret template that sets the type, or replace with registry federation where the platform supports it
   kubernetes.io/tls	Migrate if a person created it; leave alone if a certificate controller issues and renews it
   kubernetes.io/service-account-token, bootstrap.kubernetes.io/token	Do not migrate; replace legacy long-lived token Secrets with projected tokens (see How to rotate a Kubernetes service account token)
   Any Secret with an ownerReference to another controller	Leave to that controller; ESO will report a conflict rather than take it over

   Also record, for each Secret, which Pods consume it and how: env or envFrom, a volume, or a subPath mount. That decides whether a rotation needs a restart.
3. Install the delivery mechanism and connect it to the store with workload identity. Create a namespaced SecretStore per team (or a ClusterSecretStore only where a central team owns all paths) that authenticates with the cluster's workload identity, and grant that identity read access only to the store paths for that namespace. A store credential kept in another Kubernetes Secret recreates the problem you are solving. See the Kubernetes Workload Identity Checklist.
4. Seed the external store. Two options:
   • Push from the cluster: a PushSecret (apiVersion: external-secrets.io/v1alpha1) selects the existing Secret with spec.selector.secret.name, maps each secretKey to a remoteRef.remoteKey and property, and writes it to the stores in spec.secretStoreRefs. Leave deletionPolicy at its default None so deleting the PushSecret later does not delete the remote value. The store identity needs write access for this step only; remove it afterwards.
   • Generate new values in the store: for credentials you can rotate at the source (database users, API keys), create a fresh value directly in the secrets manager. This combines migration and rotation in one step and is the better option where the old value is known to be in Git.
5. Create the ExternalSecret alongside the existing Secret. For the first pass write to a new target name so you can compare without touching the running workload:

   apiVersion: external-secrets.io/v1
   kind: ExternalSecret
   metadata:
     name: app-db
     namespace: payments
   spec:
     refreshPolicy: Periodic
     refreshInterval: 1h0m0s
     secretStoreRef:
       name: payments-store
       kind: SecretStore
     target:
       name: app-db-migrated
       creationPolicy: Owner
     data:
       - secretKey: password
         remoteRef:
           key: prod/payments/app-db
           property: password

   Confirm the ExternalSecret reaches condition Ready with reason SecretSynced, and that key names and type match the original. Use target.template to set the Secret type and any fixed keys the application expects.
6. Cut the workload over. Either point the Deployment at the new Secret name, or take over the original name. To take over the original, set target.name to it with creationPolicy: Orphan for the cut-over period. ESO adopts an existing Secret that has no ownerReference and starts updating it, and with Orphan the Secret survives if you later delete the ExternalSecret. Restart Pods that read the Secret as environment variables. Run the application's own health checks, then move to creationPolicy: Owner once the change has been stable for an agreed period.
7. Remove the old source. Delete the Secret manifest or Helm values entry from the repository and from CI variables, so the next deploy does not overwrite the synced Secret with the old value. If values were committed, the copies remain in Git history; see How to Remove Secrets From Git History, and rotate regardless.
8. Rotate, then tighten RBAC. Rotate each migrated value in the store and confirm it reaches the Pods. Then remove human get, list and watch on Secrets in migrated namespaces where nobody needs them: the Kubernetes documentation warns that granting list on Secrets implicitly lets the subject fetch their contents. See the Kubernetes RBAC Guide.

Refresh settings that matter

Verify it worked

• Sync status: every ExternalSecret is Ready with reason SecretSynced and a recent status.refreshTime.
• No manual Secrets left: re-run the inventory; each in-scope Secret now has an ExternalSecret owner or is on the documented exceptions list.
• Rotation test: change a non-critical value in the store and confirm it reaches the Pod within the refresh interval (plus a restart for environment variables).
• Repository clean: secret scanning on the repositories that held the manifests reports no live values. See the Git Secret Scanning Guide.

Rollback and common errors

With creationPolicy: Orphan, rollback is to delete the ExternalSecret: the Secret stays with its last synced value, and you can restore the old manifest if needed. With Owner, deleting the ExternalSecret garbage-collects the Secret, so switch back to Orphan first. Avoid taking plaintext exports of Secrets as a rollback backup; if you need a copy, the external store is the copy.

Small estate or large organisation

A single cluster with a few applications can migrate namespace by namespace with one SecretStore each and a short checklist. Large estates should treat this as a programme: generate the inventory across clusters automatically, publish a naming convention and path policy for the store, give each platform team a namespaced store bound to its own identity, migrate by wave (non-production, then low-risk production, then shared credentials), and use an admission policy to block new manually created Secrets in migrated namespaces. See the Kubernetes Admission Control Guide and, for the wider programme, the Secrets Management Guide.

Detection and measurement

ESO exposes Prometheus metrics including externalsecret_sync_calls_total, externalsecret_sync_calls_error and externalsecret_status_condition; alert on a rising error ratio and on ExternalSecrets that are not Ready. In the cluster, Kubernetes audit logs should show Secret reads only from the controller, kubelets and expected workloads; alert on list or get of Secrets by humans in migrated namespaces (see the Kubernetes Audit Log Detection Guide). In the store, alert on reads by identities other than the cluster's.

• Migration coverage: in-scope Secrets backed by an ExternalSecret or CSI volume, as a percentage of the inventory.
• Rotated after migration: migrated values rotated at least once since cut-over (target 100%).
• Sync health: ExternalSecrets not Ready for more than one refresh interval (target zero).
• Repository exposure: secret scanning findings in deployment repositories (target zero live findings).
• Static store credentials: SecretStores authenticating with a static key rather than workload identity (target zero).
• Human Secret access: users with get, list or watch on Secrets in migrated namespaces.

Common failure patterns

• Moving the value without rotating it. The old password is still in Git history and old etcd backups. Migration only reduces risk once the value changes.
• A static cloud key to reach the store. The Tesla Kubernetes incident shows how cloud credentials stored in a cluster become the attacker's next step; a static key for the secrets manager is the same risk with a bigger prize.
• One ClusterSecretStore for everything. Any namespace can then request any path the store identity can read. Prefer namespaced stores or restrict which namespaces may use a cluster store.
• GitOps fighting the operator. The old Secret manifest stays in the repository and the GitOps tool re-applies it after every sync, reverting the value. See the GitOps Security Guide.
• Forgetting environment variable consumers. The store rotates, the Secret updates, and the application keeps the old value until a restart that nobody scheduled, then fails at the worst time.

Framework mapping

Practitioner checklist

• Inventory Secrets by name, type, keys and owner without decoding values.
• Exclude service account tokens, bootstrap tokens and controller-owned Secrets.
• Choose ESO, CSI driver or direct reads per application, and record why.
• Authenticate the operator or provider to the store with workload identity, never a static key.
• Seed the store with PushSecret or fresh values, then remove the store write permission.
• Cut over with creationPolicy: Orphan, restart environment variable consumers, then switch to Owner.
• Delete old manifests and CI variables, and clean Git history where values were committed.
• Rotate every migrated value and confirm delivery.
• Remove human Secret read rights in migrated namespaces and block new manual Secrets with admission policy.
• Alert on ESO sync errors, unexpected Secret reads and store reads by other identities.

Standards and references

• Kubernetes: Secrets
• Kubernetes: Good practices for Kubernetes Secrets
• Kubernetes: Managing Secrets using kubectl
• External Secrets Operator: ExternalSecret
• External Secrets Operator: PushSecret
• External Secrets Operator: Lifecycle, ownership and deletion policy
• Secrets Store CSI Driver: Sync as Kubernetes Secret
• External Secrets Operator: Metrics

Related NHI Mgmt Group resources: External Secrets Operator security guide, Kubernetes Secrets Security Guide, Secrets Management Guide, Kubernetes NHI Security Guide, HashiCorp Vault Guide.