Once a secret lives in an external store, the hard question is how it gets into the pod, and the three common delivery patterns put the plaintext, the store credential and the rotation logic in very different places. This guide compares a per-pod sidecar agent, a one-shot init container and the Secrets Store CSI Driver as delivery mechanisms, each with its threat model, architecture, failure modes, controls and a worked scenario. It assumes you have already decided to use an external secrets manager; that decision, and native Kubernetes Secrets with encryption at rest, are covered in Kubernetes Secrets vs External Secret Stores. It is written for platform engineers, Kubernetes security owners and the teams that review their admission policies. Where products are named as examples they are listed alphabetically, and the order is not a ranking.
Key takeaways
- The short answer: use a sidecar when secrets are short-lived or dynamic and the application can reload files, an init container when a value only needs to be present at start, and the CSI Driver when you want no extra container per pod and a node-level, volume-based model.
- All three can keep values out of etcd. The difference is where the plaintext sits on the node (a shared memory volume, a tmpfs mount, or a Kubernetes Secret written by the injector) and which process holds the credential to the store.
- Rotation is the real separator. An init container fetches once and never again; a sidecar renews and re-renders while the pod runs; the CSI Driver refreshes mounts only if its alpha rotation feature is turned on.
- The injector is a privileged trust anchor. A mutating webhook or a root DaemonSet decides which pods receive which secrets, so its configuration, RBAC and failure policy deserve the same review as the secrets manager itself.
- Environment variables undo most of the benefit. Whatever the delivery pattern, converting a file into an env var brings back leakage through process listings, crash dumps and child processes, and removes live rotation.
What each pattern is
All three patterns answer the same question: which component authenticates to the secrets manager on behalf of a pod, and where does it write the result? They differ in lifetime (once, for the life of the pod, or per mount) and in location (inside the pod or on the node).
Sidecar agent
A secrets agent runs as an extra container in the same pod for the whole pod lifetime. It authenticates, fetches, renders secrets to a shared in-memory volume and keeps renewing leases and re-rendering files when values change. HashiCorp Vault Agent Injector is a common example: a mutating admission webhook intercepts pod CREATE and UPDATE events, and when a pod carries the annotation vault.hashicorp.com/agent-inject: "true" it adds an agent sidecar and a shared memory volume mounted at /vault/secrets. CyberArk Secrets Provider for Kubernetes has a sidecar mode that polls Conjur at an interval set by the conjur.org/secrets-refresh-interval annotation and updates the target when values change.
Since Kubernetes v1.33 the native sidecar feature is stable: a sidecar is declared as an entry in initContainers with restartPolicy: Always, which gives it ordered startup before the application and keeps it running alongside. Older injectors place the agent in containers, where start order is not guaranteed.
Init container
An init container runs to completion before the application containers start. It authenticates, writes secrets to a shared volume (or to a Kubernetes Secret) and exits. The application never sees a running secrets client. Vault Agent Injector runs an init container by default (vault.hashicorp.com/agent-pre-populate defaults to true), and the agent-pre-populate-only annotation removes the sidecar so only the init container remains. CyberArk Secrets Provider has an init container mode that populates Kubernetes Secrets before the application starts; its documentation states that rotation and update detection are not supported in that mode.
Secrets Store CSI Driver
The Secrets Store CSI Driver is a Kubernetes SIG project that runs as a DaemonSet. When a pod with a CSI inline volume referencing a SecretProviderClass starts, the driver calls a provider plugin over gRPC on a Unix domain socket, the provider fetches from the external store, and the driver writes the content into a volume mounted into the pod. The kubelet may pass the pod's service account token to the driver, which forwards it to the provider so the provider can authenticate as the pod. There is no extra container in the pod. Provider and configuration detail is in Kubernetes Secrets vs External Secret Stores; this page focuses on how it compares as a delivery mechanism.
Side-by-side comparison
| Property | Sidecar agent | Init container | Secrets Store CSI Driver |
|---|---|---|---|
| Where the store client runs | Inside each pod, for the pod's lifetime | Inside each pod, only before start | On each node (driver DaemonSet and provider plugin) |
| Who authenticates to the store | The pod's service account, via the agent | The pod's service account, via the init container | The provider, usually using the pod's service account token passed by the kubelet |
| Where plaintext lands | Shared in-memory volume in the pod | Shared volume in the pod, or a Kubernetes Secret if configured | tmpfs mount on Linux nodes; node filesystem on Windows |
| Rotation while running | Yes, renews and re-renders | No; new values need a pod restart | Only with alpha auto rotation enabled (off by default) |
| Dynamic, leased secrets | Good fit; the agent renews leases | Poor fit; leases expire unrenewed | Possible with provider support; renewal depends on the provider |
| Extra resource cost | A container per pod (Vault Agent memory limit defaults to 128Mi) | Brief, at start only | One driver and provider per node |
| Admission component | Mutating webhook | Mutating webhook, or plain manifest | None; the volume is declared in the pod spec |
| Node privilege required | None beyond the pod | None beyond the pod | Driver runs as root in a privileged pod with hostPath and bidirectional mount propagation |
| Behaviour if the store is down at start | Pod may start without secrets unless the agent gates startup | Pod waits; init container retries | Mount fails, so the pod does not start |
| Short-lived Jobs | Native sidecars stop when the Job's main containers finish; legacy sidecars may hold the Job open | Natural fit | Natural fit |
Threat models
Each pattern removes some exposure and adds its own. The table maps the main threats to the pattern most affected and the ATT&CK technique that describes the attacker behaviour.
| Threat | Most affected pattern | Why | ATT&CK |
|---|---|---|---|
| Secret readable on node after compromise | CSI Driver, then all | Every mounted secret for every pod on the node sits in node-managed tmpfs; a container escape reaches them all | T1611 Escape to Host, T1552.001 Credentials In Files |
| Secret persisted to disk | CSI Driver on Windows, any pattern with swap | The CSI project states Windows writes to the node filesystem, and swap can move tmpfs pages to disk | T1552.001 Credentials In Files |
| Env var leakage | Any pattern that exports to env | Values are copied into process environments, child processes and diagnostic dumps | T1552 Unsecured Credentials |
| Injector manipulation | Sidecar and init container (webhook) | Whoever can change the webhook or annotations controls what is injected where | T1610 Deploy Container |
| Pod impersonation to obtain secrets | All | Anyone who can create a pod with a given service account receives that account's secrets | T1552.007 Container API |
| Stale credential after revocation | Init container | No component refreshes or removes the value until the pod restarts | T1552 Unsecured Credentials |
- Sidecar threat model: the agent holds a renewable store token for the pod's life, so a compromised application container in the same pod can usually read both the rendered files and, if exposed, the token. Vault Agent only writes the token to the shared volume when
agent-inject-tokenis set (default false); keep it that way unless the application needs to call the store directly. The other new risk is the webhook: its service runs with rights to mutate pod specs cluster-wide. - Init container threat model: the smallest runtime footprint, because nothing in the pod holds a store credential after start. The trade is staleness: a revoked or rotated secret stays on the volume until the pod is replaced, and if the init container writes to a Kubernetes Secret, the value is back in etcd and visible to anyone with get on Secrets in that namespace.
- CSI Driver threat model: no store client inside the pod and no webhook, but a privileged root DaemonSet on every node and provider plugins that act on behalf of any pod scheduled there. Node compromise is the concentrated risk. Enabling sync to Kubernetes Secrets (the
secretObjectsfeature) also puts the value back in etcd.
Rotation behaviour in practice
Rotation needs two things: something that fetches the new value, and an application that notices. Teams often solve the first and forget the second.
| Pattern | How new values arrive | What the application must do |
|---|---|---|
| Sidecar agent | Agent re-renders files when the value or lease changes; CyberArk sidecar mode compares SHA-256 checksums at each refresh interval | Watch the file or re-read it per use; Conjur sentinel files (CONJUR_SECRETS_UPDATED, from v1.4.1) can trigger a reload |
| Init container | Only when a new pod starts | Nothing; rotation is a rollout, so rotation tooling must restart the workload |
| CSI Driver | With --enable-secret-rotation=true, the kubelet republishes the volume and the driver re-fetches, no more often than --rotation-poll-interval (default 2m) | Watch the mounted files; env vars sourced from a synced Secret need a restart |
The Kubernetes documentation notes that a volume mounted with subPath does not receive updates, which applies to native Secret volumes and catches teams who mount a single key that way. For dynamic database or cloud credentials, a sidecar that renews leases is the natural fit; see Dynamic Secrets and Ephemeral Credentials Guide. When rotation causes outages, the usual cause is an application that cached the old value; Secret Rotation Outage Troubleshooting Guide covers diagnosis.
Failure modes
- Sidecar race at startup: with a legacy sidecar in
containers, the application can start before secrets are rendered. Vault Agent Injector avoids this by default because the init container pre-populates; CyberArk documents the same risk for its sidecar mode withenvFromand suggests sentinel files or an init container. Native sidecars with astartupProberemove the race. - Init container ordering: if another init container needs secrets (a migration step, for example), the secrets init container must run first. Vault provides
agent-init-first(default false) for this. - Webhook outage: if the injector service is unavailable, pods either fail admission or, with a permissive failure policy, start without injection. Know which one your configuration produces.
- Store unavailable at mount: CSI fetch happens on pod start and restart, so a store outage blocks scheduling of new pods and recovery of crashed ones. Init containers retry and delay startup in the same way. Sidecars with pre-populate behave like init containers at start, then tolerate brief outages while running.
- Silent rotation gap: CSI rotation is off by default and alpha; teams assume mounts refresh and keep revoked values in pods for days.
- Resource pressure: per-pod sidecars at scale add memory and connections to the store. Vault's comparison page describes the injector as a dedicated client per pod and the CSI provider as one client per node.
Controls for every pattern
- Bind store roles to dedicated service accounts. Vault advises against binding roles to the default service account. Scope each store role to one namespace and service account; see Kubernetes Service Account Token Guide.
- Control who can create pods with secret-bearing service accounts. Use admission policy to restrict which namespaces may set injection annotations or reference a given SecretProviderClass. See the Kubernetes Admission Control Guide.
- Mount secrets only into containers that need them. The Kubernetes good practices page says so directly; Vault's
agent-inject-containerslimits which containers get the volume. - Prefer files over env vars, and make the application watch or re-read them. The trade-offs are covered in Environment Variables vs Secrets Manager.
- Keep swap off on nodes that host secrets (the kubelet
failSwapOndefault is true) and treat Windows nodes using CSI as persisting secrets to disk. - Protect the injector: restrict who can edit MutatingWebhookConfiguration objects, the injector Deployment, the CSI DaemonSet and provider pods; alert on changes in the audit log. See the Kubernetes Audit Log Detection Guide.
- Use short-lived authentication. Projected, audience-bound service account tokens are the bootstrap credential in all three patterns; that is how you avoid a secret zero. See the Secrets Zero Problem Guide.
- Revoke on shutdown where possible. Vault's
agent-revoke-on-shutdown(default false) revokes the agent token when the pod ends, shrinking the window for leased credentials.
Choosing: decision guidance
| Situation | Fit | Reason |
|---|---|---|
| Dynamic database or cloud credentials with leases | Sidecar | Something must renew leases and re-render while the pod runs |
| Static config secret read once at boot | Init container | No running client; rotation is handled by rollout |
| Batch Jobs and CronJobs | Init container or CSI | No long-running agent to hold the Job open; native sidecars also work |
| Very high pod density, strict per-pod overhead budget | CSI Driver | One client per node instead of per pod |
| Multi-tenant nodes with weak isolation | Sidecar or init container | Avoids a single privileged node component holding secrets for every tenant's pods |
| Windows node pools | Review carefully | The CSI project states Windows mounts may persist to the node filesystem |
| No mutating webhooks allowed by policy | CSI Driver, or explicit init container in the manifest | Neither depends on runtime mutation |
| Third-party chart expecting env vars | Any, with a sync step | Accept that the value is in a Kubernetes Secret, then harden etcd and RBAC |
Small organisations with one or two clusters usually do best with a single pattern applied everywhere, typically an init container plus restart-based rotation, because it is the easiest to reason about. Large organisations commonly run two: a sidecar for dynamic credentials and the CSI Driver or init containers for static ones. Record the pattern per namespace so reviewers know where the plaintext lives and how rotation is expected to reach it.
Worked scenario
A payments platform runs an API service, a nightly reconciliation CronJob and a reporting service on a shared cluster, all reading from one external store.
- API service: uses dynamic database credentials with a one hour lease. A sidecar renews the lease and re-renders the connection file; the application's connection pool re-reads the file when it changes. The agent token is not written to the shared volume, and revoke-on-shutdown is on.
- Reconciliation CronJob: needs a static partner API key. An init container fetches it, the Job runs and exits, and nothing stays running. Rotation of the partner key is followed by no action, because the next Job run picks up the new value.
- Reporting service: a vendor chart that only accepts environment variables. The team uses a sync to a Kubernetes Secret, confirms encryption at rest with KMS v2, restricts get on Secrets in that namespace and schedules a rolling restart after each rotation.
- Review: admission policy limits injection annotations to these three namespaces, and the audit log alerts on any change to the webhook configuration.
The result is three patterns in one cluster, each chosen for its rotation need rather than by habit, with the exceptions written down.
Detection and measurement
- Audit events to alert on: changes to MutatingWebhookConfiguration objects, SecretProviderClass objects, the CSI driver DaemonSet and injector Deployment; pods created with injection annotations outside approved namespaces.
- Store-side logs: authentication by role and service account; reads from unexpected namespaces; a sudden rise in authentications per pod, which often means a crash loop or a misconfigured refresh.
- Node signals: privileged container starts other than approved system components; hostPath mounts of kubelet pod directories.
Metrics that show the model is working:
- Share of workloads using an approved delivery pattern, by namespace.
- Share of secrets delivered as files rather than env vars.
- Median time from secret rotation in the store to the new value in use by every pod.
- Number of pods still holding a revoked credential 24 hours after revocation.
- Count of store roles bound to default service accounts (target zero).
Framework mapping
| Framework | Reference | How it applies |
|---|---|---|
| NIST SP 800-53 Rev 5 | IA-5 Authenticator Management | Rotation and lifetime of delivered secrets |
| NIST SP 800-53 Rev 5 | AC-6 Least Privilege | Per service account store roles, per container mounts |
| NIST SP 800-53 Rev 5 | SC-28 Protection of Information at Rest | tmpfs versus disk, swap, sync to etcd |
| NIST CSF 2.0 | PR.AA-05 | Access permissions managed with least privilege |
| NIST CSF 2.0 | PR.DS-01 | Confidentiality of data at rest on nodes and in etcd |
| MITRE ATT&CK | T1552.001, T1552.007, T1611 | Credential theft from files, from the container API, and after escape to host |
Where identity fits
In all three patterns the pod's Kubernetes service account is the identity the secrets manager trusts, so these are non-human identity decisions as much as storage ones. An unused or over-shared service account becomes a route to secrets regardless of delivery pattern. The Kubernetes NHI Security Guide covers inventory and ownership, and the longer-term option of removing stored secrets in favour of federation is covered in Workload Identity Federation Guide.
Practitioner checklist
- Record, per namespace, which delivery pattern is used and where plaintext lands.
- Match pattern to rotation need: sidecar for leased secrets, init container or CSI for static ones.
- Confirm whether CSI auto rotation is enabled, and if not, document restart-based rotation.
- Bind every store role to a dedicated service account and namespace, never the default account.
- Restrict injection annotations and SecretProviderClass use through admission policy.
- Keep agent tokens off shared volumes unless the application needs them.
- Deliver secrets as files and make applications reload them; avoid subPath mounts for rotating values.
- Keep swap disabled and review Windows node pools that use the CSI Driver.
- Alert on changes to webhooks, CSI DaemonSets and provider plugins.
- Measure time from rotation to adoption and pods holding revoked credentials.
Standards and references
- Kubernetes: Good practices for Kubernetes Secrets
- Kubernetes: Sidecar Containers
- Kubernetes SIGs: Secrets Store CSI Driver concepts
- Kubernetes SIGs: Secrets Store CSI Driver secret auto rotation
- CyberArk: Secrets Provider for Kubernetes, init container and sidecar modes
- HashiCorp: Vault Agent Injector
- HashiCorp: Vault Agent Injector annotations
- HashiCorp: Kubernetes integration comparisons
Related NHI Mgmt Group resources: Kubernetes Secrets Security Guide · External Secrets Operator security guide · HashiCorp Vault Guide · CyberArk Security Hardening Guide · Secrets Management Guide