Kubernetes Secrets vs External Secret Stores

Kubernetes Secrets are the default way to give a pod a credential, but they are stored unencrypted in etcd unless you configure otherwise and are readable by anyone who can create a pod in the namespace, so most teams end up asking whether to keep secrets in the cluster or in an external store. This guide compares the options as a decision: native Secrets with encryption at rest, an operator that syncs from an external store into native Secrets, and the Secrets Store CSI Driver that mounts external secrets straight into pods. It sets out what each protects against, where each still exposes the value, when to choose which, and how the patterns combine. It is for platform engineers and Kubernetes security owners. Hardening of native Secrets is covered in the Kubernetes Secrets Security Guide and the operator itself in the External Secrets Operator security guide; this page does not repeat them.

Key takeaways

It is three options, not two. Native Secrets only, external store synced into native Secrets, and external store mounted through a CSI volume each put the plaintext in different places.
Syncing does not remove the native Secret. An operator that copies values into Kubernetes Secrets keeps the external store as the source of truth, but the copy in etcd still needs encryption at rest and tight RBAC.
Encryption at rest is a baseline, not a choice. The Kubernetes documentation lists identity (no encryption) as the default provider and recommends KMS v2 when using a third-party key management tool.
The CSI Driver keeps secrets out of etcd only if you leave sync off. Its sync-as-Kubernetes-Secret and auto rotation features are listed as alpha in the project documentation.
Pod creation is the real boundary. In every model, whoever can create a pod with the right service account can read the secret, so namespace isolation and admission control matter as much as where the secret is stored.

The three models

1. Native Kubernetes Secrets

A Secret object is stored by the API server in etcd and delivered to pods as environment variables or as files in a volume. The Kubernetes documentation states that Secrets are, by default, stored unencrypted in etcd, that anyone with API access or etcd access can read or modify them, and that anyone authorised to create a Pod in a namespace can read any Secret in that namespace, including indirectly through a Deployment. Individual Secrets are limited to 1MiB. Values in manifests are base64 encoded, and the good practices page states plainly that base64 is not an encryption method.

2. External store, synced into native Secrets

An operator such as External Secrets Operator (ESO) reads from an external secrets manager and writes the values into ordinary Kubernetes Secrets. ESO defines SecretStore and ClusterSecretStore for the connection to a provider and ExternalSecret for what to fetch. Its API reference describes refreshInterval (a value of zero creates the Secret once and does not update it), refreshPolicy values CreatedOnce, Periodic (the default) and OnChange, and creation and deletion policies for the target Secret. Pods consume the result exactly as they consume any native Secret, so applications need no change.

3. External store, mounted through the Secrets Store CSI Driver

The Secrets Store CSI Driver (secrets-store.csi.k8s.io) is a Kubernetes SIG project. The Kubernetes good practices page describes it as a DaemonSet that lets the kubelet retrieve secrets from external stores and mount them as a volume into specific Pods that you authorise. A SecretProviderClass custom resource names the provider and objects; the pod declares a CSI inline volume. The project lists providers for Akeyless, AWS, Azure, Conjur, GCP, OpenBao and Vault (listed alphabetically, as on the project page), and supports Linux and Windows containers. Syncing mounted content into a Kubernetes Secret and auto rotation are listed as alpha features.

Side-by-side comparison

Property	Native Secrets	External store synced to Secrets	External store via CSI Driver
Source of truth	The cluster (often a Git manifest)	External secrets manager	External secrets manager
Plaintext in etcd	Yes unless encryption at rest is on	Yes, as the synced copy; encryption at rest still needed	No, unless the alpha sync feature is enabled
Delivery to pod	Env var or volume	Env var or volume (unchanged for apps)	CSI volume only; env var only through synced Secret
Rotation	Manual update; volumes update eventually, env vars at restart, `subPath` never	Operator refresh per `refreshInterval`; then native update behaviour	Auto rotation is alpha; otherwise at pod start
Audit of reads	Kubernetes audit log for API reads	External store logs operator reads; Kubernetes audit logs pod-side reads	External store logs each mount fetch
Cross-cluster and non-Kubernetes reuse	No	Yes, same secret serves many clusters and VMs	Yes
Dependency at pod start	API server only	None once synced	External store must be reachable when the pod mounts
Main new risk	etcd and RBAC exposure	Operator's store credential and cluster-wide RBAC	Driver DaemonSet privileges and provider auth per pod

Encryption at rest: the baseline for any model that writes Secrets

If Secrets exist in the cluster, from manifests or from an operator, configure an EncryptionConfiguration on the API server. The Kubernetes encryption task page lists the providers below. The first provider listed encrypts new writes and all are tried in order for decryption.

Provider	What the Kubernetes docs say	Use
`identity`	Default; no confidentiality	Only as the last entry during migration
`aescbc`	Weak; not recommended because of CBC padding oracle vulnerability; key on control plane host	Avoid
`aesgcm`	Must be rotated every 200,000 writes; not recommended without automated rotation; key on host	Only with automated rotation
`secretbox`	Strong (XSalsa20 and Poly1305); key on host; may not be acceptable where high levels of review are required	Self-managed clusters without KMS
`kms` v1	Deprecated since v1.28 and disabled by default since v1.29	Migrate off
`kms` v2	Stable since v1.29; envelope encryption; a good choice with a third-party key management tool	Preferred

Two points from the same page matter for the comparison. Local keys in the configuration file protect against an etcd compromise but not a control plane host compromise, so KMS keeps the key outside the cluster. And encryption applies only on write: existing Secrets stay unencrypted until rewritten, which the docs handle by replacing all Secrets in place. Managed Kubernetes services document their own options for this; see How to check Kubernetes secrets encryption at rest.

Choosing: decision guidance

Situation	Fit	Reason
Single cluster, small team, no existing secrets manager	Native Secrets with KMS v2 or provider-managed encryption, strict RBAC, secrets kept out of Git	Lowest operational load; acceptable when the cluster is the only consumer
Secrets shared by several clusters, VMs or functions	External store synced by an operator	One source of truth, rotation in one place, apps unchanged
Requirement that secret values never sit in etcd	CSI Driver without sync	Values go from the store to a pod volume only
Third-party charts that expect a native Secret or env var	Operator sync	CSI mounts alone cannot populate env vars
Strong availability need at pod start	Operator sync, or native Secrets	Pods start from the synced copy even if the store is briefly unavailable
GitOps where everything must be in Git	External store references in Git (ExternalSecret or SecretProviderClass), or encrypted manifests	Git holds a pointer or ciphertext, not base64 plaintext; see SOPS and Sealed Secrets
Short-lived database or cloud credentials	External store issuing dynamic secrets, or no secret via workload identity	Lease-based expiry rather than stored values

Small organisations usually get most value from turning on encryption at rest, removing secrets from Git and tightening who can create pods, before adding an operator. Large organisations with many clusters gain more from an external store as the system of record, a standard per namespace (operator or CSI), and central audit of reads. Mixing both models in one cluster is common; record per namespace which is in use so reviewers know where the plaintext lives.

Security trade-offs in detail

Operator blast radius: a sync operator usually has cluster-wide rights to write Secrets and holds a credential to the external store. A ClusterSecretStore serves every namespace, so a namespace that can create an ExternalSecret may be able to pull any path that store can read. Scope stores per tenant; the External Secrets Operator security guide covers this.
CSI Driver node exposure: the driver runs as a DaemonSet on every node and the kubelet performs the mount, so node compromise exposes mounted secrets for pods on that node, as it does for native Secret volumes.
Provider authentication: both external models must authenticate to the store. Prefer workload identity (cloud workload identity or a federated service account token) to a static store token kept in a Kubernetes Secret, which would bring back the problem the external store was meant to solve. See the Kubernetes Service Account Token Guide.
Env var delivery: whichever model you choose, delivering the value as an environment variable means it is fixed at container start and visible to every process in the container. Prefer volume mounts; see Environment Variables and .env Files.
Drift and deletion: with ESO the default deletion policy Retain keeps the Kubernetes Secret if the provider data is deleted, so revoking at the source does not remove the copy. Revoke the credential at the target system, not only in the store.

How they work together

The patterns are layers. A common mature design: an external secrets manager is the system of record and issues short-lived credentials where it can; namespaces that host third-party software use an operator to sync into native Secrets, with KMS v2 encryption at rest; namespaces with sensitive in-house services use the CSI Driver without sync; workload identity authenticates both to the store; and admission control blocks raw Secret manifests with data in Git-managed namespaces. Kubernetes audit logs and the store's audit logs together show every read.

How to implement: phased approach

Baseline: enable encryption at rest (KMS v2 or the managed service's envelope encryption), rewrite existing Secrets so they are encrypted, and restrict get, list and watch on Secrets. The good practices page warns that list access lets the subject read Secret contents.
Inventory: list Secrets by namespace, type and consumer, and find those defined in Git manifests.
Choose per namespace: operator sync or CSI mount, using the decision table.
Set up store authentication with workload identity, one store or role per tenant.
Pilot: migrate one non-critical application. Issue a new credential in the store rather than copying the old value, and run old and new side by side; How to migrate Kubernetes secrets to an external Secrets Manager walks through the steps.
Test rotation: rotate in the store and confirm the pod sees the new value within the expected window (operator refresh then volume update, or pod restart).
Roll out by ring and remove the old Secret manifests from Git; rewrite Git history where real values were committed.
Roll back by re-pointing the deployment to the previous Secret only with a freshly issued credential, never the revoked one.

Detection and measurement

Monitor: Kubernetes audit events for get, list and watch on secrets, especially list across namespaces and reads by human users; creation of pods or workloads that mount Secrets they did not use before; changes to ExternalSecret, SecretStore, ClusterSecretStore and SecretProviderClass objects; external store audit logs for reads by the operator or driver identities from unexpected clusters. The good practices page suggests audit rules that alert on concurrent reading of multiple Secrets by a single user. See the Kubernetes Audit Log Detection Guide.

Metrics:

Clusters with encryption at rest verified (target 100%).
Percentage of application secrets whose source of truth is the external store.
Count of Secret manifests with data in Git repositories (target zero).
Subjects with list or watch on Secrets cluster-wide.
Sync failures and time since last successful refresh per ExternalSecret.
Median credential age for secrets consumed by pods.

Common failure patterns

Base64 treated as encryption: Secret manifests committed to Git. The Exposed Kubernetes Secrets 2023 page describes registry credentials in dockercfg and dockerconfigjson Secrets found in public repositories, including a key reaching a large artefact repository.
External store adopted, etcd left unencrypted: the operator writes plaintext copies into etcd, and backups of etcd carry them.
Static store token in a Secret: the operator or CSI provider authenticates with a long-lived token stored in the cluster.
One ClusterSecretStore for everyone: any namespace that can create an ExternalSecret can pull other teams' secrets.
Rotation assumed, not tested: values delivered as env vars or through subPath never update, so rotation breaks the application or is quietly disabled.
Broad pod creation rights: developers who cannot read Secrets can still create a pod that mounts them.

Framework mapping

Framework	Reference	Relevance
MITRE ATT&CK	T1552.007 Container API	Reading credentials through the Kubernetes or container API
MITRE ATT&CK	T1552.001 Credentials In Files	Secret values in manifests and mounted files
NIST SP 800-53 Rev 5	SC-28(1) Cryptographic Protection; SC-12 Cryptographic Key Establishment and Management	Encryption at rest and KMS key handling
NIST SP 800-53 Rev 5	AC-6 Least Privilege; IA-5 Authenticator Management; AU-12 Audit Record Generation	RBAC on Secrets, credential lifecycle, audit of reads
NIST CSF 2.0	PR.DS-01; PR.AA-05; PR.PS-04	Data at rest, least privilege, log records
ISO/IEC 27001:2022 Annex A	A.5.17 Authentication information; A.8.24 Use of cryptography	Secret handling and encryption

Practitioner checklist

Verify encryption at rest on every cluster and that existing Secrets were rewritten.
Use KMS v2 or the managed service's envelope encryption, not aescbc.
Remove list and watch on Secrets from everyone except system components.
Decide per namespace: native, operator sync or CSI mount, and record it.
Authenticate operators and CSI providers with workload identity, not stored tokens.
Scope secret stores per tenant; avoid one cluster-wide store for all teams.
Prefer volume mounts to environment variables; avoid subPath for secrets that rotate.
Test rotation end to end before relying on it.
Keep only pointers or ciphertext in Git.
Alert on bulk Secret reads and changes to store and provider objects.

Kubernetes Secrets vs External Secret Stores: Encryption, Sync and CSI Driver Options