Secrets in Container Images and Build Artefacts Guide: Find, Prevent and Revoke Baked-In Credentials

Container images and other build artefacts routinely ship with live credentials inside them, and deleting the file in a later build step does not remove it. A cloud key passed as a build argument, a .env file swept up by COPY . ., or a token compiled into a bytecode cache stays readable by anyone who can pull the image, for as long as the image exists in any registry or cache. This guide explains how secrets get into image layers, metadata and attestations, how to build images that never contain them, how to scan registries and artefact stores for the ones already there, and how to respond when you find one. It is for platform, DevOps and application security engineers and for NHI owners responsible for the credentials involved. Docker Hub account and access token controls are covered separately in the Docker Hub Tokens Security Guide; this guide is about the content of the artefacts.

Key takeaways

• Layers are append-only. Under the OCI image specification a deletion in a later layer is a whiteout entry that hides the file; the original bytes remain in the lower layer and can be extracted.
• Build arguments and ENV are not secret. Docker documents that both persist in the final image, and SLSA provenance in mode=max exposes build argument values.
• Use secret and SSH mounts. RUN --mount=type=secret and --mount=type=ssh make a credential available to one instruction only and never write it to a layer or to provenance.
• Scan the artefact, not just the repository. Secrets that never reached Git, such as compiled caches, local config files and generated files, only show up when you scan image layers and packaged artefacts.
• Removing the image is not remediation. Copies live in caches, mirrors and attackers' pulls; revoke and rotate the credential first, then clean up the artefact.

How secrets end up in images and artefacts

Most leaks come from a short list of build habits. Knowing which one produced a finding tells you which control failed.

OWASP's CICD-SEC-6 Insufficient Credential Hygiene names this directly: build-only credentials left in container image layers, and the need to remove secrets from all artefacts, including image layers, binaries and Helm charts. See the Top 10 CI/CD Security Risks.

Why deleting the file does not work: layers, history and attestations

• Layer changesets: the OCI image specification defines each layer as a tar changeset applied on top of the layers below. A removed path is recorded as an empty whiteout file named with a .wh. prefix (or an opaque .wh..wh..opq for a whole directory). The whiteout hides the file in the running container; the earlier layer still contains it and anyone who pulls the image can unpack that layer.
• Image configuration and history: ENV values are stored in the image configuration, and the history records the instructions that built each layer. Tools that show image history reveal secrets passed on the command line of a RUN instruction.
• Provenance attestations: Docker's SLSA provenance in mode=min excludes build argument values and secret identities; mode=max exposes build argument values and includes a copy of the Dockerfile. Docker states that secret mounts never appear in provenance. If you publish max-mode provenance, any secret in a build argument is published with it.
• Squashing is not a fix: squashing layers into one can remove the lower copy from that image, but earlier pushed tags, build caches, CI caches and registry mirrors still hold the original layers. Treat squashing as clean-up after rotation, never as the remediation.
• Caches and mirrors: BuildKit caches, CI cache exports (for example registry cache images), pull-through caches and replicated registries all keep copies. Public images may already have been pulled by scanners; the Docker Hub Secrets Leak research found that most secrets removed from images had never been revoked.

Building images that never contain secrets

Secret and SSH mounts

• Secret mounts: pass the secret with docker build --secret id=npm,src=$HOME/.npmrc (or env= for an environment variable) and consume it with RUN --mount=type=secret,id=npm,target=/root/.npmrc npm ci. The default mount path is /run/secrets/<id>, and the env= option exposes it as an environment variable for that instruction only.
• SSH mounts: for cloning private Git dependencies, forward the agent with docker buildx build --ssh default and use RUN --mount=type=ssh. The private key never enters the build context.
• Remote contexts: when building from a private Git repository, BuildKit supports the predefined GIT_AUTH_TOKEN and GIT_AUTH_HEADER secrets for fetching the context, rather than embedding a token in the URL.
• Build checks: enable Docker build checks in CI and fail on SecretsUsedInArgOrEnv, which flags ARG and ENV keys whose names suggest sensitive data. It is name-based, so it is a guard rail, not a scanner.

Context and stage hygiene

• Allow-list the build context: write .dockerignore as deny-all followed by explicit includes (for example * then !src/, !package.json). COPY and ADD can reach any file in the context, so a deny-list always misses something new.
• Multi-stage builds: do dependency installs and compilation in a builder stage, then copy only the built output into a minimal final stage. Secrets mounted in the builder stage, and caches created there, do not reach the final image.
• Exclude caches and local files: __pycache__, *.pyc, node_modules/.cache, IDE folders, .git, .env*, *.pem, *.key and cloud CLI folders belong in the ignore file even if you allow-list.
• Build from a clean checkout in CI: images built on developer laptops pick up local files. Publish only images built by the pipeline from a reviewed commit, and sign them so the registry can tell them apart.

Runtime injection instead of baked configuration

Images should be identical across environments and carry no credentials. Supply secrets at runtime through the orchestrator or a secrets manager: Kubernetes Secrets or an external secrets operator, cloud task definitions that reference a secrets manager entry, or a sidecar or agent that fetches short-lived credentials. Better still, give the workload its own identity (IAM roles for service accounts, managed identities, workload identity federation) so there is no static secret to inject. See the Kubernetes Secrets Security Guide and the Workload Identity Federation Guide.

Scanning registries and artefact stores

Repository scanning finds secrets in source; artefact scanning finds the ones that never reached Git. You need both, at three points.

• Layer coverage: confirm the scanner inspects every layer, not just the flattened file system, so whited-out files are found. Trivy, for example, scans image layers for secrets by default but skips layers it identifies as the base image for speed; check with debug output when a known secret is missed.
• Binary and compiled files: choose scanning that looks inside compiled and binary files where your stack produces them. Trivy documents detection in compiled Python .pyc files; the PyPI Admin GitHub Token Leak was a token inside a .pyc file in a public image, missed because the ignore file did not exclude bytecode.
• Rules and allow lists: start with built-in rules for cloud keys, Git platform tokens, private keys and AI provider keys; add custom rules for internal token formats. Review allow lists: an allowed path silently hides anything inside it.
• Validity checks: where the scanner or a follow-up tool can test whether a credential is live, use it to prioritise, but treat every finding as exposed until rotated.
• Coverage inventory: list every registry and artefact repository, who can push to it, and whether it is scanned. See the Container Registry Security Guide and the Secrets Scanning Buyer's Guide.

Responding to a secret found in an image

1. Revoke or rotate first: disable the credential at its issuer before touching the image. Deleting a tag does nothing for copies already pulled. See the Leaked Credential Response Playbook.
2. Scope the exposure: record which images, tags and digests contain it, whether the repository was public, when the first affected image was pushed, and pull counts or registry access logs since then.
3. Check for use: search the issuer's audit logs for activity by that credential from unexpected sources across the whole exposure window, not just recent days.
4. Fix the build: identify the route (build argument, broad copy, cache, hardcoded value) and change the Dockerfile, ignore file or code so the next build is clean.
5. Purge artefacts: rebuild and push clean images, delete affected tags and digests from every registry, mirror and cache repository, and invalidate build caches.
6. Prevent recurrence: add a scanner rule or build check for the pattern and record the finding against the credential's owner.

Rolling out controls

1. Baseline: scan all images and artefacts currently in your registries, prioritising public and production repositories, and rotate everything found.
2. Warn mode in CI: add image secret scanning and Docker build checks to pipelines in report-only mode for two to four weeks, tune rules and allow lists, and fix the Dockerfiles that trigger findings.
3. Block on new findings: switch to failing builds for high-confidence rules (cloud keys, private keys, Git platform tokens), with a documented exception process that expires.
4. Standardise builds: publish base Dockerfile templates with secret mounts, allow-list ignore files and multi-stage structure; require signed images from the pipeline for production admission.
5. Registry continuous scanning: scan on push and re-scan on a schedule as rules change.

Rollback: if blocking stops releases, revert that rule to warn mode for the affected repository only, with an owner and a date, rather than disabling scanning globally.

Decision guidance: small teams with one registry can rely on a single scanner in CI plus the registry's built-in scanning and a strict ignore-file template. Large organisations with many registries, artefact types and contractors need a central inventory of registries, scheduled scanning across all of them, admission control that accepts only signed pipeline images, and monitoring of public registries for images under the organisation's name.

Detection and measurement

• Credential use from unexpected places: alert when a credential that should only be used by a workload inside your network is used from elsewhere, the usual sign that a baked secret has been harvested (ATT&CK T1552.001 Credentials In Files, followed by T1078.004 Cloud Accounts for cloud keys).
• Registry events: log pushes from non-pipeline identities, visibility changes from private to public, and unusual pull volumes on repositories that contain sensitive images.
• Tampered images: unexpected pushes to internal base images can indicate T1525 Implant Internal Image; signature verification at admission catches them.

Framework mapping

Common failure patterns

• Delete-after-copy: copying a key or config file in, using it, and removing it in a later RUN, believing the image is clean.
• COPY . . with a thin ignore file: the whole working directory, including .env and .git, lands in a layer; this was a leading route in the Docker Hub Secrets Leak.
• Uncommitted local secrets: a value never committed to Git but present on the build machine, as in the PyPI Admin GitHub Token Leak, where a highly privileged token sat in a public image for about 15 months.
• Repository-only scanning: relying on Git secret scanning and never scanning images or artefact repositories.
• Takedown without rotation: deleting the image or making the repository private and leaving the credential valid.
• Max-mode provenance with build-argument secrets: publishing detailed attestations that disclose the values the team thought were transient.

Practitioner checklist

• Ban secrets in ARG and ENV; enable Docker build checks and fail on SecretsUsedInArgOrEnv.
• Use --mount=type=secret and --mount=type=ssh for every build-time credential.
• Adopt allow-list .dockerignore files and multi-stage builds with minimal final stages.
• Exclude bytecode, caches, .git, .env* and key files from every build context.
• Scan every layer of every image before push, and re-scan registries and artefact repositories on a schedule.
• Inject secrets at runtime or use workload identity; keep images free of environment-specific configuration.
• On a finding, revoke first, then scope, check use, fix the build and purge all copies.
• Monitor public registries for images published under your organisation or by contractors.

Standards and references

• Open Container Initiative: Image Layer Filesystem Changeset
• Docker Docs: Build secrets
• Docker Docs: SecretsUsedInArgOrEnv build check
• Docker Docs: Provenance attestations
• OWASP: Top 10 CI/CD Security Risks
• Trivy: Secret scanning
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: Docker Hub Tokens Security Guide · Container Image Security Guide · Terraform and IaC Pipeline Credential Security Guide · OWASP NHI Top 10 NHI2 Secret Leakage Guide · Secrets Management Guide