Kubernetes Secrets and ESO: closing the secret lifecycle gap

At a glance

What this is: This guide explains how External Secrets Operator syncs external secret values into Kubernetes while keeping application interfaces unchanged.

Why it matters: It matters because IAM, NHI, and workload identity teams need a consistent way to govern secret lifecycle, reduce drift, and avoid treating Kubernetes Secrets as a complete control plane.

By the numbers:

• The project is hosted under the external-secrets GitHub organisation and has accumulated over 4,000 stars and hundreds of contributors, making it one of the most actively maintained Kubernetes ecosystem projects for secrets management.
• 15–30 minutes is the recommended refresh interval for, secrets that rotate frequently, such as database credentials and short-lived tokens.

👉 Read Akeyless's guide to syncing Kubernetes Secrets with External Secrets Operator

Context

Kubernetes Secrets solve distribution, not governance. They are base64-encoded objects stored in etcd, so teams that rely on them alone still have to solve encryption, rotation, auditability, and lifecycle control elsewhere. That is why the real identity problem here is not secret delivery into pods, but how to govern non-human credentials across the workload lifecycle.

External Secrets Operator addresses that gap by letting Kubernetes consume standard Secret objects while the actual values live in a dedicated secrets manager. For IAM and NHI programmes, that means policy moves to the source of truth, not to application code, and the cluster becomes a consumer rather than the system of record.

Key questions

Q: How should security teams govern Kubernetes Secrets in multi-tenant clusters?

A: Treat Kubernetes Secrets as a delivery mechanism, not the system of record. Use namespace-scoped SecretStores, limit provider read paths to the minimum required, and keep cluster-wide sharing exceptional. That approach preserves tenancy boundaries while still allowing standard Secret consumption in workloads.

Q: Why do Kubernetes secret rotation projects fail if pods are not reloaded?

A: Because changing the Secret object does not guarantee the application reads the new value. If the workload only loads credentials at startup, the updated secret sits unused until a restart or file re-read occurs. Rotation must therefore include consumption behaviour, not just backend update logic.

Q: What do teams get wrong about External Secrets Operator refresh intervals?

A: They often treat refreshInterval as a generic polling setting instead of a control tied to credential lifetime and provider load. If the interval is too long, stale credentials remain active. If it is too short, backend rate limits and operational noise increase without improving governance.

Q: Which frameworks are most relevant for Kubernetes secret governance?

A: NIST Cybersecurity Framework 2.0 and the OWASP Non-Human Identity Top 10 are the most direct fits because this problem spans protection, access control, and credential lifecycle. In practice, teams should use them to align secret storage, rotation, and audit logging with the cluster’s identity model.

Technical breakdown

How the External Secrets Operator reconciliation loop works

External Secrets Operator is a Kubernetes controller that watches ExternalSecret resources, resolves the referenced SecretStore, fetches the remote secret, and materialises a native Kubernetes Secret. The important design point is reconciliation, not push delivery: the controller polls on refreshInterval and updates the target object when the provider value changes. Status conditions such as SecretSynced and SecretSyncedError expose whether the last sync succeeded. This pattern preserves application compatibility, but it also means the secret source of truth and the in-cluster copy can diverge between polling cycles.

Practical implication: Set alerting on sync failures and define a rotation interval that is shorter than the secret TTL.

SecretStore, ClusterSecretStore, and namespace isolation

SecretStore defines how ESO authenticates to the external provider and is namespace-scoped, while ClusterSecretStore is cluster-wide and reusable from any namespace. That distinction matters because it changes the blast radius of a misconfiguration. A namespaced store keeps access paths local to the team that owns them, but a cluster-scoped store can expose shared paths across tenants if RBAC is too broad. In practice, ESO is not just a sync mechanism. It is an access boundary that should mirror the cluster’s tenancy model and the provider’s path structure.

Practical implication: Use namespaced SecretStores by default and reserve ClusterSecretStore for genuinely shared secrets.

Why rotation and pod restarts are separate problems

ESO updates the Kubernetes Secret object, but Kubernetes does not restart consuming pods when that object changes. That means secret rotation is only half the job. If an application reads its credential once at startup, the new value will sit unused until a rollout or reload occurs. Teams usually pair ESO with a reload controller or an application that re-reads mounted files. The key architectural point is that secret freshness, application reload behaviour, and provider TTL are distinct control layers and must be designed together.

Practical implication: Link secret rotation to a pod refresh mechanism before you rely on short-lived credentials in production.

Threat narrative

Attacker objective: The objective is to obtain usable credentials from the cluster path and extend their usefulness beyond the intended lifecycle.

1. Entry occurs when a secret is written into etcd as a Kubernetes Secret object, which creates a persistent in-cluster copy of a credential value.
2. Credential exposure follows if that Secret is reused widely, copied into GitOps flows, or left unrotated after the upstream value changes.
3. Impact is stale or overexposed credentials remaining usable longer than intended, increasing the chance of unauthorized access across workloads.

Breaches seen in the wild

• MongoBleed breach — MongoBleed exposed secrets across 87K MongoDB servers.
• Shai Hulud npm malware campaign — Shai Hulud campaign: npm malware exposed secrets on GitHub.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Base64-encoded Kubernetes Secrets are not a governance control. They are a transport format for application compatibility, not a lifecycle boundary for credentials. The real failure is assuming that storing a secret in a native Kubernetes object makes it managed. It does not, which is why NHI programmes need external source-of-truth controls, not just in-cluster copies.

Secret lifecycle control breaks when rotation and consumption are decoupled. ESO can update the object, but if the workload does not reload, the credential change does not actually reach the application. This creates a hidden governance gap between policy intent and runtime effect. Practitioners should treat secret rotation as a two-part control, not a single event.

Namespaced secret stores define the operational trust boundary better than cluster-wide sharing. A ClusterSecretStore is convenient, but convenience can collapse tenancy boundaries if access paths and RBAC are not tightly scoped. The field-level lesson is that workload identity governance must follow namespace ownership, not administrative convenience.

Ephemeral secret delivery does not eliminate the need for auditability. The value of externalising secrets is not only shorter exposure windows, but also a defensible record of who fetched what and when. That makes the NHI lifecycle auditable in a way native Kubernetes Secrets alone cannot support, which is why external provider logs should be part of the control set.

Secret sprawl is now a workload identity problem, not just a vault problem. Once secrets are propagated into clusters, governance shifts from storage hygiene to entitlement scope, refresh cadence, and runtime consumption. The practical conclusion is that teams need to manage the entire path from provider to pod as one identity lifecycle.

From our research:

• 91% of former employee tokens remain active after offboarding, leaving organisations vulnerable to potential security breaches, according to The 2025 State of NHIs and Secrets in Cybersecurity.
• 62% of all secrets are duplicated and stored in multiple locations, causing unnecessary redundancy and increasing the risk of accidental exposure.
• Follow the lifecycle angle: The NHI Lifecycle Management Guide shows how provisioning, rotation, and offboarding need to operate as one control plane, not three disconnected tasks.

What this signals

Secret lifecycle discipline will keep moving from the vault into the runtime. ESO-style patterns are becoming the bridge between external secrets platforms and cluster workloads, which means governance teams need to think in terms of entitlement scope, refresh cadence, and reload behaviour together. The practical signal is that secret management is now a workload identity issue, not a storage-only problem.

When 62% of all secrets are duplicated and stored in multiple locations, the control objective shifts from secrecy to consistency. That figure from our research points to a larger operational truth: the more copies exist, the harder it becomes to know which credential version is live in the cluster. Teams should expect drift detection and lifecycle verification to matter as much as encryption.

Secret sprawl is a lifecycle failure as much as a technical one. Once the cluster can consume external values, the next maturity step is to make offboarding, rotation, and audit traceability visible in the same governance workflow. That is where the Guide to the Secret Sprawl Challenge becomes useful for practitioners deciding what to prioritise next.

For practitioners

• Default to namespace-scoped SecretStores Use SecretStore per team namespace and only promote to ClusterSecretStore when the same secret genuinely serves multiple tenants. Keep provider paths narrow and match them to namespace ownership so a cluster-wide store does not become a hidden shared trust domain.
• Shorten refresh intervals to match real TTLs Set refreshInterval based on the actual lifetime of the upstream secret, not the convenience of a default. For frequently rotated credentials, poll more often than the token or password expiry so the cluster does not continue serving stale values after the provider rotates.
• Pair sync with a pod reload mechanism Assume that an updated Kubernetes Secret will not be consumed automatically by every application. Use a reload controller or built-in file re-read logic so secret rotation reaches the workload before the old credential loses validity.
• Alert on sync failure conditions Treat SecretSyncedError as an operational control failure, not a benign status. Route it to on-call, correlate it with provider audit logs, and verify that the last successful sync still matches the current provider value.
• Separate static secrets from dynamic credentials Use dynamic secrets for databases, cloud tokens, and other credentials that benefit from short-lived issuance, and reserve static values for cases where the application cannot tolerate runtime renewal. That distinction reduces the chance of long-lived credential persistence in the cluster.

Key takeaways

• Kubernetes Secrets alone do not solve credential governance because they preserve application compatibility rather than enforce secret lifecycle control.
• ESO improves security only when teams also align refresh intervals, namespace boundaries, audit logging, and pod reload behaviour with the upstream secret’s lifetime.
• The main governance gain is not just secret delivery, but a clearer identity lifecycle model for how credentials move from provider to workload.

Key terms

• ExternalSecret: An ExternalSecret is a Kubernetes custom resource that tells the operator what secret to fetch from an external provider and how to project it into a native Secret object. It separates secret intent from secret storage, which lets teams keep application contracts stable while moving governance to the backend store.
• SecretStore: A SecretStore defines how Kubernetes authenticates to an external secrets provider within a namespace. It contains the connection and credential reference needed for sync, so it acts as the local trust boundary for a team or application namespace.
• ClusterSecretStore: A ClusterSecretStore is the cluster-scoped version of a SecretStore and can be referenced from any namespace. It simplifies shared access, but it also broadens the blast radius if access paths or RBAC are not tightly controlled.
• Refresh Interval: Refresh interval is the polling cadence ESO uses to check the external provider for secret changes. It is a governance setting as much as an operational one, because it determines how long a rotated credential can remain stale in the cluster before reconciliation updates it.

What's in the full article

Akeyless's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step YAML manifests for SecretStore, ClusterSecretStore, and ExternalSecret setup
• Provider-specific authentication details for Akeyless Kubernetes JWT, AWS IAM, Azure AD, and GCP
• Production tuning guidance for refreshInterval values and failure handling
• Audit and restart integration patterns for secrets that must be reloaded at runtime

👉 Akeyless's full guide covers setup, refresh timing, and production pitfalls for ESO

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.