Infrastructure as code pipelines hold some of the most powerful non-human credentials in an organisation, and Terraform quietly copies many secrets into state and plan files that are far less protected than the vault they came from. A single pipeline role that can create IAM users, change network rules and read every database password is a prize for any attacker who reaches the runner, the state bucket or a saved plan artefact. This guide covers the three places credentials live in a Terraform or OpenTofu workflow: the state and plan files, the provider credentials the pipeline uses to call cloud APIs, and the secrets that resources consume. It explains what sensitive, ephemeral values and write-only arguments actually change, how to replace static cloud keys with dynamic, per-run credentials, and how to detect misuse. It is for platform engineers, cloud security teams and NHI owners, and goes deeper than the CI/CD Pipeline Identity Security Guide.
Key takeaways
- State is a secrets store you did not choose. Terraform writes resource attributes, including generated passwords, private keys and values read from data sources, into state and plan files in plain form inside the file.
sensitive = trueonly hides output. HashiCorp documents that sensitive values are still stored in state and plan files, andterraform output -jsonor-rawprints them in plain text.- Ephemeral values and write-only arguments keep secrets out of storage. Ephemeral resources and variables (Terraform 1.10 and later) and write-only
_woarguments (Terraform 1.11 and later) are never persisted to state or plan. - Static cloud keys in pipeline variables are the biggest avoidable risk. Use OIDC workload identity (HCP Terraform dynamic provider credentials or CI OIDC federation) with trust conditions that match the full subject, and separate plan and apply roles.
- Protect the state backend like a vault. Encrypt with a customer-managed key, version the bucket, restrict read access to the pipeline identity, and log every read of state objects.
Where credentials live in an IaC workflow
Credential exposure in Terraform is rarely one mistake: several stores each hold a copy of something sensitive, and each needs a different control.
| Location | What ends up there | Who can usually read it | Primary control |
|---|---|---|---|
State file (terraform.tfstate, remote backend object) | All managed resource attributes: generated passwords (random_password), TLS private keys (tls_private_key), access keys created by aws_iam_access_key, values read through data sources | Anyone with read on the backend, every pipeline that uses terraform_remote_state | Encrypted backend, narrow read access, ephemeral and write-only patterns |
Saved plan file (terraform plan -out) | Planned values, including sensitive ones, and backend configuration passed with -backend-config | Anyone who can download CI artefacts | Short artefact retention, encryption, never attach plans to pull requests |
.terraform directory | Backend configuration values supplied on the command line or hardcoded | Runner file system, cache layers | Supply backend credentials through environment variables or a partial configuration |
Variable files (*.tfvars, *.auto.tfvars) | Database passwords, API tokens, provider keys written by hand | Everyone with repository access, forever in Git history | Never commit secrets; read them at runtime from a secrets manager |
| Pipeline variables and runner environment | AWS_ACCESS_KEY_ID, ARM_CLIENT_SECRET, GOOGLE_CREDENTIALS, Vault tokens | Any step or third-party action in the job | OIDC federation, per-run credentials, job-scoped secrets |
| Logs and debug output | Values printed by TF_LOG=DEBUG, provisioners, or outputs printed with -raw | Anyone with log access | Masking, no debug logging in shared pipelines, log retention limits |
OWASP lists the same pattern as CICD-SEC-6 Insufficient Credential Hygiene in its Top 10 CI/CD Security Risks: secrets in commit history, credentials reachable by unreviewed code, secrets printed to console output, and credentials that are never rotated. See the Top 10 CI/CD Security Risks.
State and plan files: what sensitive, ephemeral and write-only really do
Terraform's own guidance is direct: if you put secret values in configuration, Terraform stores them in state and plan files. The three features that look like fixes behave very differently.
| Feature | Minimum version | Hides from CLI and UI output | Kept out of state and plan | Typical use |
|---|---|---|---|---|
sensitive = true on variables and outputs | Long available | Yes, and the sensitivity propagates to expressions that reference it | No | Stop values appearing in run logs |
ephemeral = true on variables and module outputs | Terraform 1.10 | When combined with sensitive | Yes | Passing a token into a provider block or module without persisting it |
ephemeral resource block | Terraform 1.10 | Not applicable | Yes, exists only during the operation | Reading a secret from a secrets manager, or opening a temporary connection, at run time |
Write-only arguments (password_wo with password_wo_version) | Terraform 1.11, provider support required | Not applicable | Yes, the provider receives the value and Terraform discards it | Setting database master passwords or secret values on managed resources |
- Use the ephemeral pattern for new code: read the secret with an ephemeral resource from the secrets manager, pass it to a write-only argument, and bump the
_wo_versionnumber when you rotate it. Terraform cannot detect drift on a value it never stores, so rotation is driven by that version number. - Check provider support: each provider decides which resources offer write-only arguments. Track which of your resources still need classic arguments and treat their state as secret-bearing.
- Audit existing state: pull the state for each workspace in a controlled session and search for attribute names such as
password,private_key_pem,secret,tokenandconnection_string. Any hit is a live credential that needs rotation once the code is fixed, because old state versions keep the value. - OpenTofu state encryption: OpenTofu can encrypt state and plan files client-side with
aes_gcm, using key providers such aspbkdf2,aws_kms,gcp_kmsoropenbao, configured in aterraform { encryption { } }block or theTF_ENCRYPTIONenvironment variable, withenforced = trueto refuse unencrypted writes and afallbackblock for key rotation. The project notes that encryption does not protect against the person runningtofu, so access control still matters.
Securing the state backend
Remote state is the right default, but a remote backend is only as safe as its storage controls. HashiCorp recommends storing state remotely, encrypting it at rest, restricting access and tracking access with audit logs. For the S3 backend that translates into specific settings.
- Encryption: set
encrypt = trueandkms_key_idto a customer-managed KMS key whose key policy grantskms:Decryptonly to the pipeline roles and a break-glass role. The key adds a second, separately logged access decision. - Locking: use
use_lockfile = true. DynamoDB-based locking is deprecated in the S3 backend documentation; both can be set together while you migrate. - Least-privilege permissions: the backend needs
s3:ListBucketon the bucket,s3:GetObjectands3:PutObjecton the state key, ands3:GetObject,s3:PutObjectands3:DeleteObjecton the.tflockfile. It does not need delete on the state file. - Versioning and recovery: enable bucket versioning, which the documentation strongly recommends, and remember that old versions contain old secrets. Apply lifecycle rules so non-current versions expire after your recovery window.
- Separate state per blast radius: one state per environment and per domain (network, identity, data) so a reader of the application state cannot read the identity state. Use
workspace_key_prefixand per-prefix permissions, or separate buckets for production. - Restrict
terraform_remote_state: reading another workspace's outputs requires read access to its whole state. Publish only the values consumers need through a parameter store or outputs-only mechanism instead. - Backend credentials: HashiCorp recommends supplying backend credentials through environment variables. Values passed with
-backend-configor hardcoded are written into the.terraformsubdirectory and plan files.
HCP Terraform and Terraform Enterprise encrypt state at rest and control state access through workspace permissions; the settings that matter there (state sharing, team access, agent pools) are covered in the Terraform Cloud and HCP Security Guide.
Provider credentials: from static keys to dynamic, per-run access
The credential that lets Terraform call cloud APIs is usually the most privileged NHI in the estate. Static keys stored as pipeline variables are long-lived, shared across every job that can read the variable, and hard to attribute. Both HCP Terraform and the major CI systems now support OIDC workload identity, where the platform issues a signed token per run and the cloud exchanges it for short-lived credentials.
| Approach | Credential lifetime | Attribution | Main risk | When to choose it |
|---|---|---|---|---|
| Static cloud keys in pipeline variables | Until someone rotates them | Shared, weak | Theft from runner, logs, third-party actions; key reuse outside the pipeline | Only as a temporary exception with rotation and an expiry date |
| HCP Terraform dynamic provider credentials | Per plan or apply, discarded when the run ends | Organisation, project, workspace and run phase in the token | Loose trust conditions that match only the organisation | Runs execute in HCP Terraform or Terraform Enterprise, including self-hosted agents (v1.7.0 or newer) |
| CI OIDC federation (GitHub Actions, GitLab, Azure DevOps) | Minutes to hours, set by the role session | Repository, branch or environment in the subject | Wildcard subjects that let any branch or pull request assume the role | Terraform runs inside the CI system |
| Vault-issued dynamic secrets (AWS, Azure, GCP secrets engines) | Lease-based, revocable | Vault audit log ties lease to the requesting identity | Vault token handling on the runner | Multi-cloud estates that already centralise on Vault or OpenBao |
| Runner instance identity (instance profile, managed identity) | Short-lived, automatically refreshed | Tied to the runner, not the job | Every job on that runner inherits the same rights | Dedicated, single-purpose self-hosted runners only |
Trust conditions that actually constrain access
- Match the full subject: HCP Terraform tokens carry a
subof the formorganization:ORG:project:PROJECT:workspace:WORKSPACE:run_phase:planor:apply, an issuer ofhttps://app.terraform.ioand an audience such asaws.workload.identity. HashiCorp warns that if you do not match at least the organisation name, any organisation on HCP Terraform could obtain your credentials. Match the workspace and run phase, and prefer ID claims such asterraform_workspace_idbecause names can be changed. - Split plan and apply roles: give the plan phase a read-only role (
TFC_AWS_PLAN_ROLE_ARNand the equivalents for Azure and GCP) and only the apply phase a write role. A malicious pull request that triggers a speculative plan then cannot change infrastructure. - GitHub Actions: set the audience to
sts.amazonaws.comand the subject torepo:ORG/REPO:environment:productionfor apply jobs, protected by environment approval rules. GitHub states that you must define at least one condition so untrusted repositories cannot request tokens; avoidrepo:ORG/REPO:*for any role that can write. See the GitHub Actions OIDC Federation Hardening Guide.
The wider pattern, and how to roll it out across clouds, is in the Workload Identity Federation Guide.
Scoping what the pipeline identity can do
Dynamic credentials fix lifetime, not privilege.
- One role per workspace and environment: the network workspace gets network permissions, the application workspace gets application permissions. Avoid a shared "terraform-admin" role.
- Guard identity-changing permissions: permissions such as creating IAM users, access keys, role trust policies and service account keys let a compromised pipeline create persistent access (ATT&CK T1098.001 Additional Cloud Credentials). Keep them in a separate, reviewed workspace.
- Permission boundaries and organisation policies: apply permission boundaries or service control policies so the pipeline role cannot grant more than its boundary or disable logging, whatever the code says.
- Policy as code before apply: run Sentinel, OPA or Checkov-style checks on the plan to reject hardcoded secrets, public state buckets, wildcard IAM and resources that emit secrets into state. See the Policy-as-Code for Authorisation Guide.
- Protect the code path: require reviewed merges to the branch that runs apply, pin provider and module versions with the dependency lock file, and pin third-party CI actions to a commit. A compromised module or action runs with the pipeline's credentials (T1195.002 Compromise Software Supply Chain).
Secrets consumed by resources
Terraform often needs to set secrets on resources (a database password, an API key for a SaaS provider) or create them (an access key for an application). Each choice has a different storage outcome.
- Reference, do not copy: where the target service can read a secret by reference (for example an ECS task definition pointing at a Secrets Manager ARN, or a Kubernetes external secret), pass the reference, not the value.
- Read at run time with ephemeral resources: fetch secrets from Vault, OpenBao or a cloud secrets manager with an ephemeral resource rather than a data source. A classic data source stores what it reads in state; an ephemeral resource does not.
- Do not mint long-lived keys in Terraform:
aws_iam_access_key, service account key resources and similar write the secret into state. Prefer workload identity for the application, or create the key in a secrets manager rotation workflow. See the Dynamic Secrets and Ephemeral Credentials Guide. - Provider tokens for SaaS providers: providers for Git platforms, observability tools and identity providers authenticate with API tokens. Scope them, give each workspace its own token and an owner. See the HashiCorp Vault Guide for central issuance.
Rolling it out
- Inventory: list every workspace, its backend, its provider credentials and who can read its state. Note every static key and its age.
- Scan: run secret scanning across IaC repositories and their full history, including
*.tfvarsfiles, and scan existing state files in a controlled session for secret-bearing attributes. Treat every hit as exposed and plan rotation. - Harden backends first: enable KMS encryption, versioning, public access blocks, a deny on non-TLS requests, narrow bucket and key policies, and data-event logging. This is low risk and immediately reduces exposure.
- Pilot dynamic credentials on one non-production workspace: configure the trust relationship with full subject matching and split plan and apply roles. Run plans and applies for a week alongside the old key, then remove the static key from that workspace.
- Expand by environment: move remaining non-production workspaces, then production, each with its own role. Keep the old key disabled, not deleted, for a short rollback window, then delete it.
- Refactor secret-bearing resources: move to ephemeral resources and write-only arguments as providers support them, rotate the secrets that were in state, and expire old state versions.
- Enforce: add policy-as-code rules that block new static keys, secret-generating resources outside an approved list and unencrypted backends.
Rollback: if a dynamic credential trust breaks runs, re-enable the disabled static key for that workspace only, fix the trust condition, and disable it again. Never widen the subject to a wildcard to get a pipeline working.
Small versus large organisations: a small team with one cloud account can get most of the value from an encrypted, versioned backend, CI OIDC federation and a single read-only plan role. Large estates need per-workspace roles, central policy-as-code, a platform team that owns the trust relationships, and an inventory that ties each pipeline identity to an owner.
Detection and measurement
- State access: enable object-level data events on state buckets (for AWS, CloudTrail data events for S3) and alert on
GetObjectof state keys from any principal other than the pipeline roles and break-glass role. Alert on KMSDecryptcalls for the state key from unexpected principals. - Federated sessions: alert on
AssumeRoleWithWebIdentityfor Terraform roles where the subject does not match the expected workspace or repository, or where the apply role is assumed outside a deployment window. - Use of pipeline credentials elsewhere: calls from pipeline roles or keys from IP ranges that are not your runners, or interactive console use of a pipeline role, indicate theft (T1078.004 Cloud Accounts).
- New credentials created by the pipeline: alert on access key, role trust policy and service account key changes made by the Terraform role outside the identity workspace.
| Metric | Target direction | Why it matters |
|---|---|---|
| Percentage of workspaces using dynamic or federated provider credentials | Towards 100% | Static keys are the most common source of pipeline credential theft |
| Number and maximum age of remaining static provider keys | Down, with a dated exception for each | Shows residual exposure |
| Secret-bearing attributes found in current state | Down | Measures progress on ephemeral and write-only refactoring |
| Principals with read access to production state | Small, named list | State read equals secret read |
| Trust policies with wildcard subjects | Zero for write roles | Wildcards let unreviewed branches deploy |
| Mean time to rotate a secret found in state or a repository | Hours, not weeks | Exposure window after discovery |
Framework mapping
| Framework | Reference | How it applies |
|---|---|---|
| NIST SP 800-53 Rev. 5 | IA-5 Authenticator Management | Lifetime, protection and rotation of provider keys and tokens |
| NIST SP 800-53 Rev. 5 | AC-6 Least Privilege | Per-workspace roles, read-only plan roles |
| NIST SP 800-53 Rev. 5 | SC-28 Protection of Information at Rest; SC-12 Cryptographic Key Establishment and Management | Encrypted state backends and KMS key policies |
| NIST SP 800-53 Rev. 5 | CM-2 Baseline Configuration; SA-10 Developer Configuration Management | Reviewed IaC changes, pinned providers and modules |
| NIST SP 800-53 Rev. 5 | AU-2 Event Logging | State object and KMS access logging |
| NIST CSF 2.0 | PR.AA-01, PR.AA-05, PR.DS-01, PR.PS-01, DE.CM-09 | Credential management, least privilege, data at rest, configuration management, monitoring |
| CIS Controls v8.1 | Control 3 Data Protection, Control 5 Account Management, Control 6 Access Control Management, Control 16 Application Software Security | State as sensitive data, pipeline accounts, scoped roles, secure pipelines |
| ISO/IEC 27001:2022 Annex A | A.5.17 Authentication information, A.8.4 Access to source code, A.8.9 Configuration management, A.8.24 Use of cryptography | Secret handling, repository access, IaC baselines, state encryption |
| MITRE ATT&CK | T1552.001 Credentials In Files; T1078.004 Cloud Accounts; T1098.001 Additional Cloud Credentials; T1195.002 Compromise Software Supply Chain | Secrets in state and tfvars, stolen pipeline roles, persistence via new keys, poisoned modules or actions |
| OWASP Top 10 CI/CD Security Risks | CICD-SEC-6 Insufficient Credential Hygiene | Secrets in history, logs and artefacts; temporary, scoped credentials |
Common failure patterns
- Treating
sensitiveas encryption: teams mark outputs sensitive and assume the problem is solved, while the values sit in every state version and any reader can print them withterraform output -raw. - Static admin keys in CI variables: a long-lived administrator key shared by every job is exposed to every step and third-party script. The Codecov breach showed how a tampered CI script harvested environment variables, including cloud keys, from customers' pipelines.
- Wide OIDC trust: a trust policy that matches only the organisation, or
repo:ORG/*, lets any repository or branch obtain the deployment role. - State readable by everyone in the account: broad read on the state bucket or
terraform_remote_stateshared across teams turns every developer and every pipeline into a reader of database passwords. - Plans as public artefacts: saved plan files attached to pull requests or kept for months in CI artefact storage carry the same secrets as state.
- Forgetting old versions: fixing the code but not rotating secrets that were already in state, and not expiring old state versions, leaves the exposure in place.
Practitioner checklist
- Inventory every workspace, its backend, provider credentials and state readers.
- Encrypt state with a customer-managed key, enable versioning and lockfiles, block public access and log object reads.
- Replace static provider keys with dynamic provider credentials or CI OIDC federation, matching the full subject.
- Use separate read-only plan roles and write apply roles, one per workspace and environment.
- Refactor secrets to ephemeral resources and write-only arguments; stop using data sources and resources that write secrets into state.
- Scan repositories, tfvars, CI logs and existing state for secrets and rotate every hit.
- Keep plan artefacts short-lived and never give fork pull requests credentials.
- Enforce policy-as-code rules for backends, static keys and wildcard IAM before apply.
- Alert on state reads, KMS decrypts and role assumptions outside the expected pipeline identities.
Standards and references
- HashiCorp: Manage sensitive data in your configuration
- HashiCorp: Ephemeral resources and write-only arguments
- HashiCorp: Terraform S3 backend
- HashiCorp: HCP Terraform workload identity tokens
- OpenTofu: State and plan encryption
- GitHub Docs: Configuring OpenID Connect in Amazon Web Services
- OWASP: Top 10 CI/CD Security Risks
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: CI/CD Pipeline Identity Security Guide · Terraform Cloud and HCP Security Guide · Workload Identity Federation Guide · Secrets in Container Images and Build Artefacts Guide · Secrets Management Guide