A Kubernetes Secret is a convenient place to put a credential, but out of the box it is base64-encoded text stored unencrypted in etcd and readable by anyone who can list Secrets or create a Pod in its namespace. Secrets hold the database passwords, registry credentials, TLS keys, cloud keys and API tokens that workloads use, so they are where non-human identity risk in a cluster concentrates. This guide goes deeper than our Kubernetes NHI Security Guide on one object: how Secrets are stored, who can read them, how to encrypt them with a KMS provider, when to move them to an external store, and what to log and measure. It is written for platform engineers, cluster administrators and security teams running self-managed or managed Kubernetes.
This is an independent guide from NHI Mgmt Group, based on the Kubernetes project's public documentation as of November 2026. It is not sponsored or reviewed by the Kubernetes project. Product features change, so confirm settings against the Kubernetes project's current documentation.
Key takeaways
- Base64 is not protection. The Kubernetes documentation states that base64 encoding provides no confidentiality over plain text, and that Secrets are stored unencrypted in etcd unless you configure encryption at rest.
- Three verbs and one action decide exposure. get, list and watch on Secrets, plus the right to create Pods in a namespace, all reveal Secret values. Treat every identity holding them as privileged.
- Use the KMS v2 provider for encryption at rest. It is stable from Kubernetes v1.29, uses envelope encryption with a key held outside the cluster, and means an attacker needs etcd and the KMS, not just a file on the control plane host.
- External stores reduce what the cluster holds. The Secrets Store CSI Driver mounts values from an external store into authorised Pods; the External Secrets Operator syncs them into native Secrets.
- Secret reads are invisible without an audit policy. Log Secret access at Metadata level (never Request or RequestResponse) and alert on bulk reads.
How identity and access work in Kubernetes Secrets
A Secret is a namespaced API object that the API server persists in etcd. Workloads consume it as a mounted volume, as environment variables through env[].valueFrom.secretKeyRef, or indirectly, when the kubelet uses an image pull Secret to authenticate to a registry. Every path to the value runs through a human or pipeline calling the API server, a service account, or the kubelet on the Pod's node.
- Storage: Secrets are written to etcd under
/registry/secrets/<namespace>/<name>. Without an--encryption-provider-configflag on kube-apiserver, the defaultidentityprovider applies, which the documentation states "does not provide any confidentiality protection". Anyone who can read etcd, an etcd backup or a control plane disk snapshot can read every Secret. - Delivery to nodes: a Secret is sent to a node only if a Pod on that node needs it. The kubelet writes mounted Secrets to
tmpfsso they do not reach durable storage, and deletes its copy when the Pod is deleted. A container running withprivileged: truecan read all Secrets used on its node. - Size and types: each Secret is limited to 1 MiB. Built-in types include
Opaque(the default),kubernetes.io/service-account-token,kubernetes.io/dockercfg,kubernetes.io/dockerconfigjson,kubernetes.io/basic-auth,kubernetes.io/ssh-auth,kubernetes.io/tlsandbootstrap.kubernetes.io/token. A field selector ontypefinds every registry or token Secret. - Immutability: since v1.21 a Secret can be marked
immutable: true. It can then only be deleted and recreated, which blocks unwanted updates. - Environment variables versus volumes: mounted files can be refreshed by the kubelet when the Secret changes (its default change-detection strategy is
Watch); environment variables are fixed for the life of the container and are more likely to leak through crash dumps, debug endpoints and child processes. Prefer volumes for anything that rotates.
The four ways a Secret value is exposed
| Exposure path | Who can use it | Control that closes it |
|---|---|---|
| Read through the API (get, list, watch) | Any user or service account with those verbs on secrets in the namespace or cluster | RBAC: no list or watch except system components, get only by name with resourceNames |
| Create a Pod that mounts the Secret | Anyone who can create Pods or workload controllers (Deployments, Jobs, CronJobs) in the namespace | Namespace isolation per trust boundary, admission policy, limit workload creation to pipelines |
| Read etcd or its backups directly | Anyone with etcd client certificates, backup storage or control plane host access | Encryption at rest with KMS v2, etcd peer and client TLS, backup encryption and access control |
| Leak from source control or CI | Anyone who can read the repository, manifest, Helm values file or pipeline log | No Secret manifests in Git, secret scanning that decodes base64, external stores or encrypted manifests |
The Exposed Kubernetes Secrets 2023 research found valid registry Secrets committed to public GitHub repositories, including one with access to SAP's artifact repository.
Admin roles and privileged access
RBAC in Kubernetes is additive with no deny rules, so restricting Secret access means not granting it. The Kubernetes RBAC good practices page warns that list and watch on Secrets reveal contents, not just names: a response to kubectl get secrets -A -o yaml includes every value. The Secrets good practices page sets the baseline: restrict watch and list to the most privileged system components, grant get only where normal behaviour requires it, and allow only cluster administrators to access etcd, including read-only access.
Permissions that amount to Secret access
| Permission | Why it exposes Secrets | Where it is commonly granted |
|---|---|---|
secrets: list, watch | Returns full contents of every Secret in scope | Operators, monitoring agents, backup tools, ClusterRoles with wildcards |
secrets: get without resourceNames | Reads any Secret in the namespace by name | Application roles written quickly |
pods or workload controllers: create | A new Pod can mount any Secret in the namespace and run as any service account there | The built-in edit and admin roles, CI/CD deploy identities |
pods/exec, pods/attach | Shell access to a running container reads its mounted Secrets and environment | Developer and support roles |
nodes/proxy | Reaches the kubelet API and can execute in Pods on the node, bypassing audit and admission | Monitoring tools |
escalate, bind, impersonate | Lets an identity grant itself or assume Secret read rights | Platform tooling, misconfigured admin roles |
cluster-admin or system:masters | Everything; system:masters bypasses RBAC entirely and cannot be limited by removing bindings | Break-glass kubeconfigs, bootstrap credentials left in use |
- Scope by name: an application that needs one Secret should get a Role with
verbs: ["get"]andresourceNames: ["db-credentials"], bound with a RoleBinding in its namespace. Never grant list to make a client library's discovery call work. - Isolate by namespace: the documentation recommends separate namespaces to isolate access to mounted Secrets. Put workloads with different trust levels, or different owners, in different namespaces, because anyone who can deploy into a namespace can read its Secrets.
- Review what an identity can do:
kubectl auth can-i list secrets --as system:serviceaccount:<namespace>:<name> -Aanswers the question for one identity; run it across all service accounts as part of access review. See the Kubernetes RBAC Guide. - Protect the encryption configuration: the EncryptionConfiguration file is as sensitive as etcd, because local providers keep their keys in it.
Encryption at rest: providers and rollout
Encryption at rest is configured with an EncryptionConfiguration file passed to kube-apiserver with --encryption-provider-config. The first provider in the list encrypts new writes; the others are tried in order when reading. Managed services expose this as a cluster setting instead.
| Provider | What it does | Assessment from the Kubernetes documentation |
|---|---|---|
identity | No encryption; the default | No confidentiality protection |
aescbc | AES-CBC with PKCS#7 padding, local key | Not recommended, because CBC is vulnerable to padding oracle attacks; key material sits on the control plane host |
aesgcm | AES-GCM with random nonce, local key | Must be rotated every 200,000 writes; not recommended without automated key rotation; key on the control plane host |
secretbox | XSalsa20 and Poly1305, local key | Rated strong, but uses newer algorithms that some high-review environments may not accept; key on the control plane host |
kms v1 | Envelope encryption via a KMS plugin | Deprecated since v1.28; disabled by default from v1.29 (needs --feature-gates=KMSv1=true) |
kms v2 | Envelope encryption: data encryption keys wrapped by a key encryption key held in an external KMS | Stable from v1.29; the recommended choice. Compromise needs etcd and the KMS |
Rolling out encryption without losing data
- Back up etcd first and confirm you can restore it. A wrong provider order or a lost key makes Secrets unreadable.
- Deploy the KMS plugin on every control plane node and confirm its health check. Grant the plugin's identity only encrypt and decrypt on one key.
- Write the configuration with
kms(apiVersion v2) first andidentity: {}last, so existing plain text Secrets stay readable. Scoperesourcestosecretsat minimum; add ConfigMaps or custom resources that hold sensitive data. - Enable it on one control plane node, then the rest. In high-availability clusters, each API server must be able to decrypt before any of them writes with the new key.
- Rewrite existing Secrets so they are stored encrypted:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -. - Verify in etcd: read a raw key with
etcdctl get /registry/secrets/default/<name>and check for the prefixk8s:enc:kms:v2:. Plain text JSON means the rewrite missed it. - Remove
identityfrom the read list once every Secret carries the encrypted prefix, so plain text cannot be written back by a misordered config. - Plan rotation: set
--encryption-provider-config-automatic-reload=trueso key changes are picked up (the file is polled every minute) without restarting the API server, and watch theapiserver_encryption_config_controller_automatic_reload_last_timestamp_secondsmetric.
Rollback: to back out, put identity first while keeping the KMS provider in the list for reads, rewrite all Secrets with the same replace command, and only then remove the KMS entry. Removing the KMS provider before rewriting leaves Secrets that the API server cannot decrypt.
Integrations, API tokens and service accounts
Most Secrets are credentials for something else, and the strongest control is often to stop storing them at all.
- Service account tokens: do not create Secrets of type
kubernetes.io/service-account-token; they hold non-expiring tokens. Use bound, projected tokens from the TokenRequest API. See the Kubernetes Service Account Token Guide. - Cloud credentials: replace static cloud keys in Secrets with workload identity federation, so Pods exchange a projected token for short-lived cloud credentials. The Tesla Kubernetes Cryptojacking 2018 incident involved static AWS keys in a Pod.
- Registry credentials: the documentation suggests kubelet credential providers to fetch image pull credentials dynamically instead of long-lived
dockerconfigjsonSecrets. Where pull Secrets remain, make them pull-only and scope them to one namespace. - Everything else: keep the value in an external secrets manager and bring it into the cluster with one of the patterns below.
External secret store patterns
| Pattern | How it works | Strengths | Trade-offs |
|---|---|---|---|
| Native Secrets with KMS v2 | Values created in the cluster, encrypted in etcd | No extra components; works with every tool | Values still in etcd and readable through the API; rotation is manual |
Secrets Store CSI Driver (secrets-store.csi.k8s.io) | DaemonSet; the kubelet mounts values from an external store as a CSI volume, defined by a SecretProviderClass | Values need not exist as Kubernetes Secrets; access tied to the Pod's identity at the store | Applications must read files; auto rotation and sync to Secrets are alpha features in the project |
| External Secrets Operator | Controller reads ExternalSecret resources and creates native Secrets from a SecretStore or cluster-wide ClusterSecretStore | Transparent to applications; reconciles changes from the store | Values land in etcd as normal Secrets; the operator's identity can read many store paths |
| Encrypted manifests in Git (for example Sealed Secrets or SOPS) | Values encrypted before commit, decrypted in cluster or at deploy time | GitOps-friendly; no plain text in repositories | Decryption keys become a high-value secret; still produces native Secrets |
Choosing: small teams on one managed cluster can start with native Secrets, KMS v2 and tight RBAC. Organisations that already run a central secrets manager should use the CSI driver for high-value credentials, so values never sit in etcd, and an operator for applications that only read environment variables. Limit a ClusterSecretStore to platform teams, because it lets any namespace request any path the store's identity can read. See the Cloud Secrets Managers Guide and the HashiCorp Vault Guide.
Logging and detection
Kubernetes only records Secret access if the API server runs with an audit policy (--audit-policy-file) and a log backend. The reference policy in the Kubernetes documentation logs secrets and configmaps at the Metadata level: user, verb, resource and timestamp, without request or response bodies. Never log Secrets at Request or RequestResponse, which would copy values into your log pipeline.
| Signal | Audit fields to match | Why it matters |
|---|---|---|
| Bulk Secret read | objectRef.resource=secrets, verb=list or watch, by a non-system user | The documentation names concurrent reading of many Secrets by one user as an event to alert on |
| Secret read by an unexpected identity | verb=get, user.username not in the expected set for that Secret | Stolen service account token or over-broad role (ATT&CK T1552.007) |
| Interactive access to Pods | objectRef.subresource=exec or attach | Reads mounted Secrets without touching the secrets API |
| New role grants on Secrets | create or update on roles, clusterroles, bindings that include secrets | Privilege escalation toward credential access |
| Legacy token Secrets created | create on secrets with type kubernetes.io/service-account-token | Non-expiring credential minted for persistence |
| Encryption config changes | API server restart or reload metric, file integrity on the config path | Provider order changed to identity silently disables encryption |
Forward audit logs off the control plane; on managed services, enable the control plane audit log category. Correlate with the external store's own audit log: a CSI driver or operator read shows up there, not in Kubernetes. Relevant ATT&CK techniques are T1552.007 Container API, T1552.001 Credentials In Files, T1613 Container and Resource Discovery and T1609 Container Administration Command.
Metrics that show it is working
- Encryption coverage: percentage of clusters with KMS v2 configured and 100% of sampled etcd keys carrying the
k8s:enc:kms:v2:prefix. - Secret readers: number of non-system identities with list or watch on secrets, cluster-wide and per namespace. Target zero outside named platform components.
- Externalised share: percentage of high-value credentials (database, cloud, payment, signing keys) served from an external store rather than created in cluster.
- Legacy tokens: count of
kubernetes.io/service-account-tokenSecrets. Target zero. - Secret age: median and maximum days since rotation for Secrets labelled high-value.
- Leak findings: Secret manifests found in repositories per month, and time to revoke after detection.
Hardening baseline
| Area | Baseline setting | Framework mapping |
|---|---|---|
| Encryption at rest | KMS v2 provider first, identity removed after rewrite, etcd backups encrypted | NIST SP 800-53 SC-28, SC-12; CSF 2.0 PR.DS-01; CIS Control 3 |
| Transport | TLS for API server to etcd and between etcd peers | SP 800-53 SC-8 |
| Access | No list or watch on secrets for humans or applications; get scoped with resourceNames; no system:masters in daily use | SP 800-53 AC-3, AC-6; CSF 2.0 PR.AA-05; CIS Control 6 |
| Credential lifecycle | No legacy token Secrets; workload identity for cloud; rotation dates on high-value Secrets | SP 800-53 IA-5; CSF 2.0 PR.AA-01; CIS Control 5 |
| Workload exposure | Secrets as volumes, mounted only into the container that needs them; no privileged containers (Pod Security restricted profile) | SP 800-53 CM-6 |
| Audit | Secrets at Metadata level, logs shipped off cluster, bulk read alerts | SP 800-53 AU-2, AU-6; CSF 2.0 DE.CM-09; CIS Control 8 |
Admission control enforces much of this: a policy can reject Pods that set privileged: true, Secrets of the legacy token type, or workloads that use the default service account. See the Kubernetes Admission Control Guide and, for the wider cluster baseline, the Kubernetes Security Guide.
Common misconfigurations
- Assuming a managed cluster encrypts Secrets with your key. Envelope encryption of Secrets with your own key is often a separate setting; check it.
- Granting list to make a tool work. Operators, dashboards and backup agents often ship ClusterRoles with list and watch on all Secrets. That makes their service account a cluster-wide credential store; scope or replace them.
- Committing Secret manifests or Helm values. Base64 in Git is plain text in Git. Run scanners that decode base64 in YAML; see the Git Secret Scanning Guide.
- Leaving
identityfirst after a change. A reordered provider list, or a rollback that was never finished, quietly writes new Secrets in plain text. - Logging Secrets at RequestResponse. A broad audit rule written for troubleshooting copies every Secret value into the SIEM, which then becomes the leak.
- Static cloud keys in Secrets. They never expire and work from anywhere. Move to workload identity and delete the key, which is the long-lived secret problem described in OWASP NHI Top 10 NHI7 Long-Lived Secrets Guide.
Practitioner checklist
- Confirm encryption at rest on every cluster, migrate to KMS v2, rewrite all Secrets and verify the
k8s:enc:kms:v2:prefix in etcd. - Enumerate every identity with get, list or watch on secrets, or create on Pods, and remove what is not needed.
- Scope application access to named Secrets with
resourceNamesand RoleBindings, never ClusterRoleBindings. - Delete Secrets of type
kubernetes.io/service-account-tokenafter moving their consumers to bound tokens. - Replace static cloud keys with workload identity federation and registry passwords with credential providers or pull-only tokens.
- Move high-value credentials to an external store via the Secrets Store CSI Driver or an operator, with a store identity per namespace.
- Enable an audit policy that logs secrets at Metadata level and alert on bulk reads, exec and new Secret-reading roles.
- Block Secret manifests in repositories with base64-aware scanning and an admission rule against legacy token Secrets and privileged Pods.
Standards and references
- Kubernetes: Secrets
- Kubernetes: Good practices for Kubernetes Secrets
- Kubernetes: Encrypting Confidential Data at Rest
- Kubernetes: Using a KMS provider for data encryption
- Kubernetes: Role Based Access Control Good Practices
- Kubernetes: Auditing
- Kubernetes SIGs: Secrets Store CSI Driver
- MITRE ATT&CK: Unsecured Credentials: Container API
Related NHI Mgmt Group resources: Kubernetes NHI Security Guide · Kubernetes Service Account Token Guide · Secrets Management Guide · Top 10 Kubernetes Security Risks · Azure SAS Tokens Security Guide