Join our Newsletter — 33% off our NHI Course
Home› Guides› Kubernetes Secrets Security Guide: Encryption, RBAC and External…
Guide Non-Human Identity (NHI)

Kubernetes Secrets Security Guide: Encryption, RBAC and External Secret Stores

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 8 November 2026 16 min read
On this page

A Kubernetes Secret is a convenient place to put a credential, but out of the box it is base64-encoded text stored unencrypted in etcd and readable by anyone who can list Secrets or create a Pod in its namespace. Secrets hold the database passwords, registry credentials, TLS keys, cloud keys and API tokens that workloads use, so they are where non-human identity risk in a cluster concentrates. This guide goes deeper than our Kubernetes NHI Security Guide on one object: how Secrets are stored, who can read them, how to encrypt them with a KMS provider, when to move them to an external store, and what to log and measure. It is written for platform engineers, cluster administrators and security teams running self-managed or managed Kubernetes.

This is an independent guide from NHI Mgmt Group, based on the Kubernetes project's public documentation as of November 2026. It is not sponsored or reviewed by the Kubernetes project. Product features change, so confirm settings against the Kubernetes project's current documentation.

Key takeaways

  • Base64 is not protection. The Kubernetes documentation states that base64 encoding provides no confidentiality over plain text, and that Secrets are stored unencrypted in etcd unless you configure encryption at rest.
  • Three verbs and one action decide exposure. get, list and watch on Secrets, plus the right to create Pods in a namespace, all reveal Secret values. Treat every identity holding them as privileged.
  • Use the KMS v2 provider for encryption at rest. It is stable from Kubernetes v1.29, uses envelope encryption with a key held outside the cluster, and means an attacker needs etcd and the KMS, not just a file on the control plane host.
  • External stores reduce what the cluster holds. The Secrets Store CSI Driver mounts values from an external store into authorised Pods; the External Secrets Operator syncs them into native Secrets.
  • Secret reads are invisible without an audit policy. Log Secret access at Metadata level (never Request or RequestResponse) and alert on bulk reads.

How identity and access work in Kubernetes Secrets

A Secret is a namespaced API object that the API server persists in etcd. Workloads consume it as a mounted volume, as environment variables through env[].valueFrom.secretKeyRef, or indirectly, when the kubelet uses an image pull Secret to authenticate to a registry. Every path to the value runs through a human or pipeline calling the API server, a service account, or the kubelet on the Pod's node.

  • Storage: Secrets are written to etcd under /registry/secrets/<namespace>/<name>. Without an --encryption-provider-config flag on kube-apiserver, the default identity provider applies, which the documentation states "does not provide any confidentiality protection". Anyone who can read etcd, an etcd backup or a control plane disk snapshot can read every Secret.
  • Delivery to nodes: a Secret is sent to a node only if a Pod on that node needs it. The kubelet writes mounted Secrets to tmpfs so they do not reach durable storage, and deletes its copy when the Pod is deleted. A container running with privileged: true can read all Secrets used on its node.
  • Size and types: each Secret is limited to 1 MiB. Built-in types include Opaque (the default), kubernetes.io/service-account-token, kubernetes.io/dockercfg, kubernetes.io/dockerconfigjson, kubernetes.io/basic-auth, kubernetes.io/ssh-auth, kubernetes.io/tls and bootstrap.kubernetes.io/token. A field selector on type finds every registry or token Secret.
  • Immutability: since v1.21 a Secret can be marked immutable: true. It can then only be deleted and recreated, which blocks unwanted updates.
  • Environment variables versus volumes: mounted files can be refreshed by the kubelet when the Secret changes (its default change-detection strategy is Watch); environment variables are fixed for the life of the container and are more likely to leak through crash dumps, debug endpoints and child processes. Prefer volumes for anything that rotates.

The four ways a Secret value is exposed

Exposure pathWho can use itControl that closes it
Read through the API (get, list, watch)Any user or service account with those verbs on secrets in the namespace or clusterRBAC: no list or watch except system components, get only by name with resourceNames
Create a Pod that mounts the SecretAnyone who can create Pods or workload controllers (Deployments, Jobs, CronJobs) in the namespaceNamespace isolation per trust boundary, admission policy, limit workload creation to pipelines
Read etcd or its backups directlyAnyone with etcd client certificates, backup storage or control plane host accessEncryption at rest with KMS v2, etcd peer and client TLS, backup encryption and access control
Leak from source control or CIAnyone who can read the repository, manifest, Helm values file or pipeline logNo Secret manifests in Git, secret scanning that decodes base64, external stores or encrypted manifests

The Exposed Kubernetes Secrets 2023 research found valid registry Secrets committed to public GitHub repositories, including one with access to SAP's artifact repository.

Admin roles and privileged access

RBAC in Kubernetes is additive with no deny rules, so restricting Secret access means not granting it. The Kubernetes RBAC good practices page warns that list and watch on Secrets reveal contents, not just names: a response to kubectl get secrets -A -o yaml includes every value. The Secrets good practices page sets the baseline: restrict watch and list to the most privileged system components, grant get only where normal behaviour requires it, and allow only cluster administrators to access etcd, including read-only access.

Permissions that amount to Secret access

PermissionWhy it exposes SecretsWhere it is commonly granted
secrets: list, watchReturns full contents of every Secret in scopeOperators, monitoring agents, backup tools, ClusterRoles with wildcards
secrets: get without resourceNamesReads any Secret in the namespace by nameApplication roles written quickly
pods or workload controllers: createA new Pod can mount any Secret in the namespace and run as any service account thereThe built-in edit and admin roles, CI/CD deploy identities
pods/exec, pods/attachShell access to a running container reads its mounted Secrets and environmentDeveloper and support roles
nodes/proxyReaches the kubelet API and can execute in Pods on the node, bypassing audit and admissionMonitoring tools
escalate, bind, impersonateLets an identity grant itself or assume Secret read rightsPlatform tooling, misconfigured admin roles
cluster-admin or system:mastersEverything; system:masters bypasses RBAC entirely and cannot be limited by removing bindingsBreak-glass kubeconfigs, bootstrap credentials left in use
  • Scope by name: an application that needs one Secret should get a Role with verbs: ["get"] and resourceNames: ["db-credentials"], bound with a RoleBinding in its namespace. Never grant list to make a client library's discovery call work.
  • Isolate by namespace: the documentation recommends separate namespaces to isolate access to mounted Secrets. Put workloads with different trust levels, or different owners, in different namespaces, because anyone who can deploy into a namespace can read its Secrets.
  • Review what an identity can do: kubectl auth can-i list secrets --as system:serviceaccount:<namespace>:<name> -A answers the question for one identity; run it across all service accounts as part of access review. See the Kubernetes RBAC Guide.
  • Protect the encryption configuration: the EncryptionConfiguration file is as sensitive as etcd, because local providers keep their keys in it.

Encryption at rest: providers and rollout

Encryption at rest is configured with an EncryptionConfiguration file passed to kube-apiserver with --encryption-provider-config. The first provider in the list encrypts new writes; the others are tried in order when reading. Managed services expose this as a cluster setting instead.

ProviderWhat it doesAssessment from the Kubernetes documentation
identityNo encryption; the defaultNo confidentiality protection
aescbcAES-CBC with PKCS#7 padding, local keyNot recommended, because CBC is vulnerable to padding oracle attacks; key material sits on the control plane host
aesgcmAES-GCM with random nonce, local keyMust be rotated every 200,000 writes; not recommended without automated key rotation; key on the control plane host
secretboxXSalsa20 and Poly1305, local keyRated strong, but uses newer algorithms that some high-review environments may not accept; key on the control plane host
kms v1Envelope encryption via a KMS pluginDeprecated since v1.28; disabled by default from v1.29 (needs --feature-gates=KMSv1=true)
kms v2Envelope encryption: data encryption keys wrapped by a key encryption key held in an external KMSStable from v1.29; the recommended choice. Compromise needs etcd and the KMS

Rolling out encryption without losing data

  1. Back up etcd first and confirm you can restore it. A wrong provider order or a lost key makes Secrets unreadable.
  2. Deploy the KMS plugin on every control plane node and confirm its health check. Grant the plugin's identity only encrypt and decrypt on one key.
  3. Write the configuration with kms (apiVersion v2) first and identity: {} last, so existing plain text Secrets stay readable. Scope resources to secrets at minimum; add ConfigMaps or custom resources that hold sensitive data.
  4. Enable it on one control plane node, then the rest. In high-availability clusters, each API server must be able to decrypt before any of them writes with the new key.
  5. Rewrite existing Secrets so they are stored encrypted: kubectl get secrets --all-namespaces -o json | kubectl replace -f -.
  6. Verify in etcd: read a raw key with etcdctl get /registry/secrets/default/<name> and check for the prefix k8s:enc:kms:v2:. Plain text JSON means the rewrite missed it.
  7. Remove identity from the read list once every Secret carries the encrypted prefix, so plain text cannot be written back by a misordered config.
  8. Plan rotation: set --encryption-provider-config-automatic-reload=true so key changes are picked up (the file is polled every minute) without restarting the API server, and watch the apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds metric.

Rollback: to back out, put identity first while keeping the KMS provider in the list for reads, rewrite all Secrets with the same replace command, and only then remove the KMS entry. Removing the KMS provider before rewriting leaves Secrets that the API server cannot decrypt.

Integrations, API tokens and service accounts

Most Secrets are credentials for something else, and the strongest control is often to stop storing them at all.

  • Service account tokens: do not create Secrets of type kubernetes.io/service-account-token; they hold non-expiring tokens. Use bound, projected tokens from the TokenRequest API. See the Kubernetes Service Account Token Guide.
  • Cloud credentials: replace static cloud keys in Secrets with workload identity federation, so Pods exchange a projected token for short-lived cloud credentials. The Tesla Kubernetes Cryptojacking 2018 incident involved static AWS keys in a Pod.
  • Registry credentials: the documentation suggests kubelet credential providers to fetch image pull credentials dynamically instead of long-lived dockerconfigjson Secrets. Where pull Secrets remain, make them pull-only and scope them to one namespace.
  • Everything else: keep the value in an external secrets manager and bring it into the cluster with one of the patterns below.

External secret store patterns

PatternHow it worksStrengthsTrade-offs
Native Secrets with KMS v2Values created in the cluster, encrypted in etcdNo extra components; works with every toolValues still in etcd and readable through the API; rotation is manual
Secrets Store CSI Driver (secrets-store.csi.k8s.io)DaemonSet; the kubelet mounts values from an external store as a CSI volume, defined by a SecretProviderClassValues need not exist as Kubernetes Secrets; access tied to the Pod's identity at the storeApplications must read files; auto rotation and sync to Secrets are alpha features in the project
External Secrets OperatorController reads ExternalSecret resources and creates native Secrets from a SecretStore or cluster-wide ClusterSecretStoreTransparent to applications; reconciles changes from the storeValues land in etcd as normal Secrets; the operator's identity can read many store paths
Encrypted manifests in Git (for example Sealed Secrets or SOPS)Values encrypted before commit, decrypted in cluster or at deploy timeGitOps-friendly; no plain text in repositoriesDecryption keys become a high-value secret; still produces native Secrets

Choosing: small teams on one managed cluster can start with native Secrets, KMS v2 and tight RBAC. Organisations that already run a central secrets manager should use the CSI driver for high-value credentials, so values never sit in etcd, and an operator for applications that only read environment variables. Limit a ClusterSecretStore to platform teams, because it lets any namespace request any path the store's identity can read. See the Cloud Secrets Managers Guide and the HashiCorp Vault Guide.

Logging and detection

Kubernetes only records Secret access if the API server runs with an audit policy (--audit-policy-file) and a log backend. The reference policy in the Kubernetes documentation logs secrets and configmaps at the Metadata level: user, verb, resource and timestamp, without request or response bodies. Never log Secrets at Request or RequestResponse, which would copy values into your log pipeline.

SignalAudit fields to matchWhy it matters
Bulk Secret readobjectRef.resource=secrets, verb=list or watch, by a non-system userThe documentation names concurrent reading of many Secrets by one user as an event to alert on
Secret read by an unexpected identityverb=get, user.username not in the expected set for that SecretStolen service account token or over-broad role (ATT&CK T1552.007)
Interactive access to PodsobjectRef.subresource=exec or attachReads mounted Secrets without touching the secrets API
New role grants on Secretscreate or update on roles, clusterroles, bindings that include secretsPrivilege escalation toward credential access
Legacy token Secrets createdcreate on secrets with type kubernetes.io/service-account-tokenNon-expiring credential minted for persistence
Encryption config changesAPI server restart or reload metric, file integrity on the config pathProvider order changed to identity silently disables encryption

Forward audit logs off the control plane; on managed services, enable the control plane audit log category. Correlate with the external store's own audit log: a CSI driver or operator read shows up there, not in Kubernetes. Relevant ATT&CK techniques are T1552.007 Container API, T1552.001 Credentials In Files, T1613 Container and Resource Discovery and T1609 Container Administration Command.

Metrics that show it is working

  • Encryption coverage: percentage of clusters with KMS v2 configured and 100% of sampled etcd keys carrying the k8s:enc:kms:v2: prefix.
  • Secret readers: number of non-system identities with list or watch on secrets, cluster-wide and per namespace. Target zero outside named platform components.
  • Externalised share: percentage of high-value credentials (database, cloud, payment, signing keys) served from an external store rather than created in cluster.
  • Legacy tokens: count of kubernetes.io/service-account-token Secrets. Target zero.
  • Secret age: median and maximum days since rotation for Secrets labelled high-value.
  • Leak findings: Secret manifests found in repositories per month, and time to revoke after detection.

Hardening baseline

AreaBaseline settingFramework mapping
Encryption at restKMS v2 provider first, identity removed after rewrite, etcd backups encryptedNIST SP 800-53 SC-28, SC-12; CSF 2.0 PR.DS-01; CIS Control 3
TransportTLS for API server to etcd and between etcd peersSP 800-53 SC-8
AccessNo list or watch on secrets for humans or applications; get scoped with resourceNames; no system:masters in daily useSP 800-53 AC-3, AC-6; CSF 2.0 PR.AA-05; CIS Control 6
Credential lifecycleNo legacy token Secrets; workload identity for cloud; rotation dates on high-value SecretsSP 800-53 IA-5; CSF 2.0 PR.AA-01; CIS Control 5
Workload exposureSecrets as volumes, mounted only into the container that needs them; no privileged containers (Pod Security restricted profile)SP 800-53 CM-6
AuditSecrets at Metadata level, logs shipped off cluster, bulk read alertsSP 800-53 AU-2, AU-6; CSF 2.0 DE.CM-09; CIS Control 8

Admission control enforces much of this: a policy can reject Pods that set privileged: true, Secrets of the legacy token type, or workloads that use the default service account. See the Kubernetes Admission Control Guide and, for the wider cluster baseline, the Kubernetes Security Guide.

Common misconfigurations

  • Assuming a managed cluster encrypts Secrets with your key. Envelope encryption of Secrets with your own key is often a separate setting; check it.
  • Granting list to make a tool work. Operators, dashboards and backup agents often ship ClusterRoles with list and watch on all Secrets. That makes their service account a cluster-wide credential store; scope or replace them.
  • Committing Secret manifests or Helm values. Base64 in Git is plain text in Git. Run scanners that decode base64 in YAML; see the Git Secret Scanning Guide.
  • Leaving identity first after a change. A reordered provider list, or a rollback that was never finished, quietly writes new Secrets in plain text.
  • Logging Secrets at RequestResponse. A broad audit rule written for troubleshooting copies every Secret value into the SIEM, which then becomes the leak.
  • Static cloud keys in Secrets. They never expire and work from anywhere. Move to workload identity and delete the key, which is the long-lived secret problem described in OWASP NHI Top 10 NHI7 Long-Lived Secrets Guide.

Practitioner checklist

  • Confirm encryption at rest on every cluster, migrate to KMS v2, rewrite all Secrets and verify the k8s:enc:kms:v2: prefix in etcd.
  • Enumerate every identity with get, list or watch on secrets, or create on Pods, and remove what is not needed.
  • Scope application access to named Secrets with resourceNames and RoleBindings, never ClusterRoleBindings.
  • Delete Secrets of type kubernetes.io/service-account-token after moving their consumers to bound tokens.
  • Replace static cloud keys with workload identity federation and registry passwords with credential providers or pull-only tokens.
  • Move high-value credentials to an external store via the Secrets Store CSI Driver or an operator, with a store identity per namespace.
  • Enable an audit policy that logs secrets at Metadata level and alert on bulk reads, exec and new Secret-reading roles.
  • Block Secret manifests in repositories with base64-aware scanning and an admission rule against legacy token Secrets and privileged Pods.

Standards and references

Related NHI Mgmt Group resources: Kubernetes NHI Security Guide · Kubernetes Service Account Token Guide · Secrets Management Guide · Top 10 Kubernetes Security Risks · Azure SAS Tokens Security Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 8 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.