How to find hard-coded secrets in container images: Scan every layer, triage findings, revoke what is live

A secret baked into a container image ships to every registry, cache and laptop that pulls it, and deleting the file in a later layer does not remove it. This guide is a hands-on procedure for finding those secrets: what to inspect by hand, how to run an open-source scanner across every layer, how to separate live credentials from noise, and what to do with each finding. It is for platform engineers, AppSec teams and incident responders who own registries or have been told an image is leaking. The design side, building images that never contain secrets, is covered in the Secrets in Container Images and Build Artefacts Guide; this page is the scan-and-respond runbook.

Key takeaways

• Scan the image, not just the repo. Images pick up secrets the repository never held: .env files copied by a broad COPY, build arguments, and compiled files such as Python .pyc bytecode.
• Start read-only and offline. docker history --no-trunc and docker image inspect reveal secrets in build commands and environment variables without running the image.
• Know what your scanner skips. Trivy skips layers it identifies as the base image and honours built-in allow rules; both are sensible defaults that can hide a real finding.
• Verification is a decision, not a default. Verifying a finding means presenting the credential to its provider. Do that deliberately, from a controlled host, and log it.
• Revoke, then rebuild. Removing the tag or the image does not invalidate the credential; the Docker Hub secrets leak study found most developers who removed a secret never revoked it.

When you need this

• Registry baseline: you have never scanned the images already in your registries and want to know what is in them.
• Before publishing: an image is about to go to a public registry or a customer.
• A report or alert: someone has found a credential in a public image, or a provider has notified you that one of its keys was seen.
• After a scanner change: you adopted a source-code secret scanner, which by design never sees what the build added.

The PyPI admin GitHub token leak is the reference case: the token was never committed, but compiled bytecode in __pycache__ carried it into public images because .dockerignore did not exclude *.pyc. It stayed public for roughly 15 months.

Before you start

1. Pick a scanning host. Use an isolated build or analysis machine, not a developer laptop. You will be extracting files that may contain live credentials.
2. List the images in scope. Every repository and tag, including old and untagged images that are still pullable by digest. Scan digests, not just latest.
3. Get read access to the registries with a dedicated, read-only credential. See the Container Registry Security Guide for scoping it.
4. Agree the verification rule. Decide in advance whether the team may test found credentials against providers, and who approves.
5. Prepare a findings store. Somewhere access-controlled to record image digest, layer, file path, detector and owner. Never paste raw values into tickets.

Step-by-step

Steps 1 to 5 are read-only against the image. Step 6 can contact third parties. Step 7 changes production and is destructive.

1. Read the build history. Run docker history --no-trunc IMAGE:TAG. Without --no-trunc long commands are cut off and a token in the middle of a RUN line is invisible. For a narrower view, docker history --no-trunc --format "{{.CreatedBy}}" IMAGE:TAG prints only the commands. Search the output for ARG, ENV, TOKEN, PASSWORD, KEY and URLs with credentials in them. For multi-platform images add --platform linux/amd64 (or the platform you ship).
2. Inspect the configuration. Run docker image inspect IMAGE:TAG and read the environment list and labels. A credential set with ENV is visible to anyone who can pull the image and to every container started from it.
3. Run a layer-aware scan with Trivy. trivy image IMAGE:TAG runs secret scanning by default alongside vulnerability scanning; to run secret scanning alone, use the --scanners flag with secret. Trivy scans every plaintext file plus compiled Python files against built-in rules (AWS access keys, GCP service account keys, GitHub and GitLab personal access tokens, Slack tokens and more). Use --debug to see which base image layers it skipped, and re-check those if you built your own base image. For source trees or extracted filesystems, trivy fs PATH does the same.
4. Run a second engine with verification off. TruffleHog scans images from a registry (trufflehog docker --image REPO/IMAGE:TAG), from the local daemon (--image docker://IMAGE:TAG) or from a tarball (--image file://image.tar). --image can be repeated for several images and --json gives machine-readable output. Two engines with different rule sets catch more than one.
5. Triage each finding. Record the image digest, layer, path and detector. Mark each as live production credential, test or placeholder, or false positive. Check which other tags and digests contain the same layer: a secret in a shared layer appears in every image built on it.
6. Verify only where approved (contacts providers). TruffleHog's --results=verified or --results=verified,unknown reports whether a credential worked when tested against its provider. That is useful for prioritising, but it is an authentication attempt with someone's credential. Run it from the approved host, and treat "unverified" as "unknown", not "safe".
7. Respond (destructive). For every live or unknown credential: revoke or rotate it at the provider first, then remove the image tags and digests from public and internal registries, then rebuild from a fixed Dockerfile. For GitHub tokens, follow How to revoke a leaked GitHub personal access token and check what it touched; for others, your Secrets Rotation Runbook Template.

Verify it worked

• Rescan the rebuilt image with both engines and confirm zero findings for the original detector and path.
• Confirm the old digests are gone from every registry and mirror, and that pulls of the old digest fail.
• Confirm the credential is dead at the provider: the old key shows as deleted or disabled, and provider logs show no successful use after the revocation time.
• Check the build path: docker history --no-trunc on the new image shows no secret in commands, and the Dockerfile uses RUN --mount=type=secret,id=... fed by docker build --secret id=...,src=... instead of ARG or ENV.

Rollback and common errors

The scan steps change nothing, so there is nothing to roll back. For the response, keep the previous image digest in a private, access-controlled quarantine repository until the rebuilt image is proven in production; restore from there only if the new build fails, and never re-publish it.

• Deleting the file in a new layer: the secret remains in the earlier layer. Rebuild without it.
• Scanning only latest: old tags and untagged digests stay pullable. Scan and clean by digest.
• Trusting skipped layers: if your organisation builds its own base images, Trivy's base layer skip can hide secrets in them. Scan the base image directly.
• Silencing noise with disable-rules: in trivy-secret.yaml, disable-rules takes precedence over enable-builtin-rules, so a broad disable can switch off a detector you need. Prefer allow-rules scoped to a path.
• Removing the image instead of revoking: anyone who pulled it, and every cache, still holds the value.

Hardening afterwards

• Gate the pipeline: fail the build on secret findings before push (TruffleHog's --fail returns exit code 183 when it finds valid credentials).
• Fix the Dockerfile pattern: build secrets through secret or SSH mounts, multi-stage builds so build tools and credentials do not reach the runtime stage, and a .dockerignore that excludes .env, .git, credential files and __pycache__.
• Inject at runtime: deliver secrets from a secrets manager or workload identity at start-up, never in the image; see the Kubernetes Secrets Security Guide.
• Find shadow registries: personal and contractor accounts on public registries leaked many of the Docker Hub findings. Inventory them, and govern registry tokens as described in the Docker Hub Tokens Security Guide.
• Scan on a schedule: new detectors find old secrets; rescan registries regularly.

Where identity fits: nearly every finding in an image is a non-human identity credential: a cloud access key, a service account key, a registry or package token, a CI token or an AI API key. Treat each one as an identity with an owner, a purpose and a lifetime, not as a string to delete. When you revoke, record which workload used it and replace it with the shortest-lived option that workload supports, such as workload identity federation or tokens issued at start-up by a secrets manager. If a finding has no identifiable owner, that is a second finding: an orphaned credential that nobody would have rotated. The OWASP NHI Top 10 NHI2 Secret Leakage Guide covers this class of risk across the wider estate.

Measure it: percentage of registry images (by digest) scanned in the last 30 days, live findings per 100 images, median time from finding to revocation, percentage of findings closed by revocation rather than deletion alone, and the number of pipelines with a secret-scan gate.

Practitioner checklist

• Scan every image by digest, including old and untagged images, not just latest
• Read docker history --no-trunc and docker image inspect output for secrets in commands and environment variables
• Run Trivy and TruffleHog across all layers and record digest, layer, path and detector for each finding
• Verify findings against providers only from an approved host, and treat unverified results as unknown
• Revoke or rotate every live credential at the provider before deleting the image
• Rebuild with secret mounts, multi-stage builds and a .dockerignore that excludes .env and __pycache__
• Gate pipelines on secret findings before push and rescan registries on a schedule

Standards and references

• Docker Docs: Build secrets
• Docker Docs: docker image history
• Docker Docs: Building best practices
• Aqua Security Trivy: Secret scanning
• Truffle Security: TruffleHog README
• NIST: SP 800-190 Application Container Security Guide
• MITRE ATT&CK: T1552.001 Credentials In Files

Related NHI Mgmt Group resources: Secrets in Container Images and Build Artefacts Guide · Container Registry Security Guide · Docker Hub Tokens Security Guide · How to find secrets in CI logs and build artefacts