Environment variables and secrets managers are often treated as competing places to keep a secret, when one is a delivery channel and the other is a system of record, and confusing the two is how most application credentials end up exposed. This guide answers the comparison directly: what each one does, where each one leaks, when an environment variable is still an acceptable last step, and how to combine a secrets manager with a safer delivery path such as mounted files, a sidecar or an in-process SDK call. It is for platform engineers, application owners and security architects writing the standard for how services receive credentials. The mechanics of .env files and the detail of individual products live in other guides: see the Environment Variables and .env Files guide and the Cloud Secrets Managers Guide.
Key takeaways
- They answer different questions. A secrets manager decides who may hold a secret, for how long and with what audit trail. An environment variable only decides how a process receives a value once someone already has it.
- The environment is a wide channel. OWASP notes environment variables are generally accessible to all processes and can appear in logs or system dumps, and systemd states that environment variables are not suitable for passing secrets.
- A secrets manager does not fix delivery on its own. Fetching a secret from a vault and then exporting it into the environment keeps every leak path of the environment. The win comes from changing the last mile as well.
- Rotation is where the gap shows. An environment variable is fixed at process start; a secret read from a manager or a refreshed file can change without a redeploy, and dynamic secrets can expire on their own.
- Environment variables still have a place. Non-secret configuration, local development with throwaway values, and pointers to a secret (a path or secret name rather than the value) are reasonable uses.
What each one actually is
An environment variable is a name and value pair handed to a process by whatever launches it: a shell, systemd, a container runtime, a CI runner or a serverless platform. On Linux the initial environment is passed through execve(2), and the proc_pid_environ(5) manual page says /proc/[pid]/environ holds that initial environment, with read access governed by a PTRACE_MODE_READ_FSCREDS check. Child processes inherit the environment by default. There is no access control per variable, no expiry, no versioning and no record of who read it.
The Twelve-Factor App popularised environment variables for configuration, including "credentials to external services", because they change per deploy without code changes and are unlikely to be committed by accident. That argument was about separating configuration from code, and it still holds for configuration. It predates widespread secrets managers and workload identity, and it does not address what happens to the value once it is in the process.
A secrets manager is a service that stores secrets encrypted, authenticates every caller, authorises access per secret or path, records each read and change, and supports versioning and rotation. Many can also issue dynamic secrets, which are credentials generated on request with a lease. The OWASP Secrets Management Cheat Sheet treats it as the system of record, and recommends that each caller authenticate with a platform identity rather than a static credential. For a full treatment of the discipline see the Secrets Management Guide.
The practical consequence: the choice is rarely "environment variable or secrets manager". It is "where is the secret stored and governed" and, separately, "how does the running process receive it". The two sections below treat those as separate decisions.
Side-by-side comparison
| Property | Environment variable | Secrets manager |
|---|---|---|
| Role | Delivery channel into one process | System of record with policy, audit and lifecycle |
| Access control | Anyone who can read the process environment, the unit file, the container spec or the platform console | Per secret or path, bound to an authenticated identity |
| Encryption at rest | Depends on where the value is defined (unit file, CI settings, platform store); plain in process memory | Encrypted by the service, typically with keys in a KMS or HSM |
| Audit of reads | None at the point of use | Each retrieval logged with caller identity and time |
| Rotation | Needs a restart or redeploy to pick up a new value | Versioned; consumers can re-read, and dynamic secrets expire by lease |
| Inheritance | Passed to child processes unless cleared | Only the process that calls receives the value |
| Typical leak paths | Debug pages, crash dumps, /proc, CI logs, container inspect output, support bundles | Over-broad read policy, leaked caller credential, secret copied out into env or files |
| Bootstrap need | None: the launcher already has the value | The caller must authenticate, ideally with workload identity rather than a static token |
| Operational cost | Very low | A service to run or buy, with availability planning and caching |
Where environment variable secrets leak
MITRE classifies the weakness as CWE-526, Cleartext Storage of Sensitive Information in an Environment Variable. Its observed examples include a CMS exposing environment variables in debug mode, a build plugin writing them into build XML files and a CI/CD tool logging password variables. The main paths are:
- Diagnostics: framework debug pages, error trackers and crash reporters that capture the environment, and core dumps written to disk.
- Process inspection:
/proc/[pid]/environfor anyone who passes the ptrace read check, and process listings in tools that print the environment. - Child processes: every subprocess, shell-out and plugin inherits the full environment, including third-party code the team never reviewed. systemd's manual adds that environment variables propagate across setuid and setgid executables and are exposed to unprivileged clients over D-Bus.
- Definitions at rest: unit files, Compose files, container image
ENVandARGlayers (OWASP calls out both), Kubernetes manifests in Git, and CI variable settings. - Platform consoles: serverless and PaaS consoles show function or service variables to anyone with read access to the configuration. AWS documents that Lambda encrypts environment variables at rest with a KMS key, and that with a customer managed key only users with
kms:Decrypton that key can view them in the console. - Published files:
.envfiles served from web roots. The 230 Million AWS Cloud Environments Campaign 2024 page describes attackers collecting AWS keys and other tokens from publicly reachable.envfiles and using them for extortion.
Platform stores for environment variables are a target too. The Railway Breach 2026 page covers a provider disclosing that an attacker decrypted some customers' stored environment variables after obtaining build infrastructure credentials. Encryption at rest protected the store only as far as the identities able to decrypt it.
Where a secrets manager can still leak
A secrets manager narrows exposure but moves risk to new places:
- The caller credential: if the application authenticates with a static token stored in, again, an environment variable, the manager adds a hop but no protection. This is the secret zero problem; see the Secrets Zero Problem Guide.
- Over-broad policy: one role that can read every secret in an account or a whole path turns one compromised workload into a bulk export.
- Copying out: a startup script that fetches the secret and runs
exportrecreates every leak path above. - Pipeline access: CI jobs that read production application secrets. OWASP recommends keeping application secrets out of the pipeline and letting the deployed service fetch its own at runtime with a platform identity.
- Availability coupling: if the manager is unavailable and there is no caching strategy, applications fail to start. Teams then add fallbacks that reintroduce static values.
Safer delivery patterns: the last mile
Once a secrets manager is the system of record, choose how each workload receives the value. OWASP sets out the options in its cheat sheet, and the platforms below document their own mechanisms.
| Pattern | How it works | Rotation | Fits |
|---|---|---|---|
| In-process SDK or API call | The application authenticates with its workload identity and reads the secret into memory | Re-read on a timer or on authentication failure | Services you own and can change |
| Sidecar or agent writing to an in-memory volume | A helper fetches and refreshes the secret to a shared tmpfs path the app reads | Helper refreshes; app re-reads the file | Legacy apps that read a file; Kubernetes pods |
| Short-lived init helper | A helper writes the secret to a shared volume then exits | Restart needed | Apps that read config once at start |
| Orchestrator-mounted file | The platform mounts the secret as a file, for example Docker swarm under /run/secrets/ on an in-memory filesystem, or a Kubernetes Secret volume | Kubernetes updates volume content eventually; subPath mounts do not update | Containers on an orchestrator |
| systemd credentials | LoadCredential= or LoadCredentialEncrypted= places the secret in $CREDENTIALS_DIRECTORY, read only, readable only by the service user and root, on non-swappable memory where supported | Service restart | Services on Linux hosts |
| Environment variable from a secret reference | The platform resolves a secret reference into an environment variable at start | Restart needed | Third-party images that only read env; a stepping stone |
Two notes from the documentation. systemd marks SetCredential= as unsuitable for secrets because unprivileged processes can read it over IPC, and offers SetCredentialEncrypted= for embedded values. Kubernetes warns that Secrets are stored unencrypted in etcd by default and that anyone allowed to create a Pod in a namespace can read any Secret in it; the Kubernetes Secrets Security Guide covers the hardening.
Choosing: decision guidance
| Situation | Recommended approach | Why |
|---|---|---|
| Non-secret configuration (feature flags, hostnames, log levels) | Environment variables | No confidentiality need; Twelve-Factor reasoning applies fully |
| Local development with throwaway credentials | A git-ignored .env file with non-production values | Low impact; keep production values out of developer machines |
| Production database or API credentials for a service you own | Secrets manager plus SDK call or mounted file, authenticated by workload identity | Audit, rotation without redeploy, no inheritance |
| Third-party container that reads only env | Secret reference resolved at start, short-lived or dynamic credential, process restarted on rotation | Contains exposure while the image cannot change |
| Serverless function | Secrets manager read at cold start with caching, or the platform's secret integration; avoid plain env for database credentials | AWS recommends Secrets Manager rather than Lambda environment variables for database credentials |
| CI/CD jobs | OIDC federation to the secrets manager, job-scoped reads, masked output | Removes stored CI credentials and keeps attribution |
| Cloud provider access from a workload | No secret at all: managed identity, IAM role or workload identity federation | Nothing to store, deliver or rotate |
Small organisations: a cloud provider's managed secrets service and its native integrations with functions and containers deliver most of the value with little operational overhead. Prioritise database and payment provider credentials first. Large organisations: the harder problems are policy scoping per workload, multi-cloud identity, caching and availability, and proving coverage. Set a standard that names the approved delivery patterns per platform and treat any plain environment variable secret in production as an exception with an owner and an end date.
The best outcome is often to remove the secret. Where the target supports it, use workload identity instead; see the Workload Identity Federation Guide.
How they work together
In a mature setup the secrets manager stores and governs, workload identity authenticates the caller, and the environment carries only a pointer. A typical pattern: the environment holds DB_SECRET_NAME=prod/orders/db or a file path such as DB_PASSWORD_FILE=/run/secrets/db, never the password. The application resolves the pointer at runtime. This keeps the Twelve-Factor benefit (per deploy configuration with no code change) while the value itself stays out of the environment and out of inherited process state.
How to implement: a phased migration
- Inventory: list secrets currently passed as environment variables across unit files, Compose files, Kubernetes manifests, CI settings and serverless configuration. Secret scanners and platform APIs can enumerate variable names; classify by name pattern and owner. See How to find secrets in Lambda environment variables for one platform.
- Prioritise: start with production credentials for data stores, payment and identity providers, and cloud access keys.
- Move storage: create each secret in the secrets manager, with one read policy per workload identity. Do not reuse the value: issue a new credential, so the old one can be revoked once cut over.
- Change delivery: switch the workload to an SDK call, file mount or systemd credential. Where code cannot change, use the platform's secret reference as a temporary step.
- Test: run in a staging ring with the old variable removed, confirm the application reads the new source, and confirm rotation works by rotating once before go-live.
- Cut over and revoke: deploy, verify, then revoke the old credential and delete the variable from every definition, including CI settings and image layers.
- Roll back safely: keep the previous deployment artefact ready, but roll back with a newly issued credential rather than restoring the old value, so revoked secrets stay revoked.
- Enforce: add policy checks in CI or admission control that reject manifests with literal secret values in
env, and secret scanning on repositories and images.
Detection and measurement
What to log: every secrets manager read and policy change, with caller identity and source; denied reads; reads from unexpected networks or identities. OWASP recommends recording who requested a secret, when it was used, attempts to use expired or revoked secrets, and administrative actions, with logs protected against tampering and sent to a central location.
What to watch on the environment side: secret scanner hits in CI logs, crash reports and error tracker payloads; changes to environment variable definitions in production platforms; reads of /proc/*/environ by unexpected processes where host telemetry can capture it.
Metrics:
- Percentage of production secrets stored in the secrets manager (target near 100%).
- Count of production workloads still receiving secrets as plain environment variables, trended to zero.
- Percentage of secrets manager callers authenticating with workload identity rather than a static token.
- Median secret age and rotation coverage; see How to measure secret age and rotation coverage.
- Secret scanner findings in logs, images and repositories per month.
- Number of identities able to read each high-value secret.
Common failure patterns
- Vault then export: an entrypoint script reads the secrets manager and exports values, so every child process and crash dump still carries them.
- Static token for the manager: the token that unlocks the manager is itself an environment variable with broad read access.
- Debug mode in production: error pages or diagnostic endpoints print the environment, the pattern behind CWE-526's observed examples.
- Secrets baked into images:
ENVorARGvalues left in image layers and pushed to a registry. See the Secrets in Container Images and Build Artefacts Guide. - Rotation without restart: the manager rotates, but the service holds the old value in its environment until the next deploy, so rotation silently breaks the service or is disabled.
- Web-reachable .env files: configuration files deployed into document roots, as in the 230 Million AWS Cloud Environments Campaign 2024.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE CWE | CWE-526 | Cleartext storage of sensitive information in an environment variable |
| MITRE ATT&CK | T1552 Unsecured Credentials; T1552.001 Credentials In Files | Harvesting credentials from configuration and .env files |
| MITRE ATT&CK mitigation | M1039 Environment Variable Permissions | Restricting who can modify and read environment configuration |
| NIST SP 800-53 Rev 5 | IA-5(7) No Embedded Unencrypted Static Authenticators; IA-5 Authenticator Management; SC-28(1) Cryptographic Protection; AU-2 Event Logging | Secret storage, lifecycle and audit |
| NIST CSF 2.0 | PR.AA-01; PR.AA-05; PR.DS-01 | Credentials for services managed; least privilege; data at rest protected |
| ISO/IEC 27001:2022 Annex A | A.5.17 Authentication information; A.8.24 Use of cryptography | Management of secret authentication information |
Practitioner checklist
- Write a standard that names the secrets manager as the only system of record for production secrets.
- List approved delivery patterns per platform: SDK call, mounted file, sidecar, systemd credentials.
- Allow environment variables for pointers (secret names, file paths) and non-secret configuration only.
- Authenticate every secrets manager caller with workload identity, not a stored token.
- Scope read policies to one workload per secret or path.
- Disable debug output and environment capture in production error tracking.
- Block literal secrets in manifests, Compose files and Dockerfiles with CI checks.
- Rotate once before go-live to prove the delivery path picks up new values.
- Revoke the old credential and remove the variable everywhere after cut-over.
- Track the remaining plain environment variable secrets as dated exceptions.
Standards and references
- OWASP: Secrets Management Cheat Sheet
- MITRE: CWE-526 Cleartext Storage of Sensitive Information in an Environment Variable
- systemd: systemd.exec manual (Environment and Credentials)
- Linux man-pages: proc_pid_environ(5)
- Kubernetes: Secrets
- Docker: Manage sensitive data with Docker secrets
- AWS: Securing Lambda environment variables
- The Twelve-Factor App: III. Config
Related NHI Mgmt Group resources: Secrets Management Guide · Environment Variables and .env Files · Secrets Zero Problem Guide · Dynamic Secrets and Ephemeral Credentials Guide · Leaked Credential Response Playbook