Most container vulnerability findings come from packages the application never uses, inherited from a general-purpose base image. Shells, package managers, compilers and utilities inflate scanner output, slow triage and give an intruder tools to work with after a compromise. Minimal and hardened base images remove that surface: Google's distroless images, Wolfi-based images built with apko, Alpine, and vendor hardened catalogues such as Docker Hardened Images all take different routes to the same goal. This guide compares those options, explains how to migrate with multi-stage builds, how to debug images that have no shell, how to keep them patched and verified, and what to measure. It is for platform engineers, DevSecOps teams and vulnerability managers who own container base images.
Key takeaways
- Smaller images mean better signal, not immunity. The distroless project says restricting the runtime to what the app needs "improves the signal to noise of scanners" and reduces the provenance burden; your own code and dependencies still need scanning.
- The C library is the first decision. Distroless (Debian) and Wolfi use glibc; Alpine is built around musl and busybox. Applications with native extensions may behave differently on musl.
- No shell changes operations. Use exec-form
ENTRYPOINT, debug with:debugtags or Kubernetes ephemeral containers, and stop relying onkubectl execinto production. - Freshness beats size. A tiny image that is rebuilt rarely accumulates CVEs. Pin by digest, automate digest updates and rebuild on a schedule.
- Verify what you pull. Distroless images are signed keylessly with cosign and hardened catalogues ship SBOMs, VEX and provenance. Enforce verification at build or admission time.
What "minimal" and "hardened" actually mean
- Distroless: in the project's words, images that "contain only your application and its runtime dependencies" with no package managers, shells or other programs. The smallest,
gcr.io/distroless/static-debian13, is about 2 MiB according to the project README, against about 5 MiB foralpineand 124 MiB fordebian. - Wolfi and apko: Wolfi is a Linux "(un)distribution" designed for containers, with no kernel of its own, glibc support, granular packages, a build-time SBOM for every package and a focus on the latest upstream releases. apko builds OCI images declaratively from apk packages, is reproducible by default and emits an SBOM. Chainguard sponsors Wolfi and uses it to build its curated minimal images; the Wolfi README warns that Wolfi and Alpine packages are not compatible and must not be mixed.
- Alpine: a small general-purpose distribution "built around musl libc and busybox". The Alpine project states a container needs no more than 8 MB and that all userland binaries are compiled as position independent executables with stack smashing protection. It still includes a shell and the
apkpackage manager. - Vendor hardened images: catalogues that rebuild common runtimes with a reduced footprint, non-root defaults and supply chain metadata. Docker's documentation describes Docker Hardened Images as built on Alpine and Debian with glibc and musl variants, running as non-root by default, with signed SBOMs, VEX statements and SLSA Build Level 3 provenance, and with core features free under Apache 2.0. Paid tiers add FIPS and STIG variants and a remediation SLA, per the same page.
- scratch: the empty base. Suitable for fully static binaries (for example Go with
CGO_ENABLED=0), but you must add CA certificates, time zone data and a non-root user yourself.distroless/staticexists to provide exactly those pieces.
Comparing the options
| Option | libc | Shell and package manager | Supply chain metadata | Best fit | Main trade-off |
|---|---|---|---|---|---|
| Full distro (debian, ubuntu) | glibc | Yes | Distribution signing; SBOM from your scanner | Build stages, legacy apps | Large package count and scanner noise |
| Slim variants (debian-slim) | glibc | Yes | As above | Quick win with no code change | Still a full userland |
| Alpine | musl | busybox shell, apk | Signed apk repositories | Small images where musl is tested | musl differences for native modules and some DNS or locale behaviour |
| Distroless (Debian 13) | glibc (none in static) | No (busybox only in :debug) | Keyless cosign signatures | Go, Java, Node.js, Python, Rust runtimes | Fixed runtime set; customisation via Bazel rules_distroless |
| Wolfi and apko built images | glibc | Optional; runtime images can omit both | Build-time SBOMs, reproducible builds | Teams that want to compose their own minimal images | Wolfi packages track latest versions, so plan for frequent change |
| Vendor hardened catalogue | Depends on vendor | Separate dev and runtime variants | Signed SBOM, VEX, provenance (per vendor docs) | Organisations wanting a supported catalogue and SLA | Dependence on the vendor's rebuild cadence and terms |
| scratch | None | No | Only what you add | Static binaries | You maintain certificates, users and time zone data |
Choosing a base image
- Static binary (Go, Rust with musl target):
distroless/staticor scratch. Prefer distroless static over scratch unless you have a reason to manage CA bundles and/etc/passwdyourself. - Dynamically linked native code (C, C++, Rust with glibc):
distroless/ccor a Wolfi-based glibc image. - Interpreted or managed runtimes: the matching distroless runtime (for example
java21-debian13,nodejs24-debian13,python3-debian13) or a hardened runtime image. Check native dependencies (Python wheels, Node.js add-ons) against the image's libc and library set. - Applications that genuinely need a shell at runtime: a hardened image with a shell, or Alpine, then remove that need over time. Entrypoint scripts are the usual reason, and can often be replaced with configuration or a small init binary.
- Small organisations: start with distroless or a free hardened catalogue and Renovate or Dependabot digest updates; avoid building your own distribution.
- Large organisations: publish a small set of approved "golden" base images to an internal registry, built with apko or Bazel, signed by your own pipeline, and block anything else at admission. See the Container Registry Security Guide.
Migrating with multi-stage builds
- Inventory current bases. List every
FROMline across repositories and registry images, with the scanner finding count for each base. That is your baseline for the "CVEs removed" metric. - Split build and runtime. Keep compilers and package managers in a builder stage and copy only the artefact into the runtime stage. The distroless README notes multi-stage builds need Docker 17.05 or later.
- Use exec form for entrypoints. Distroless images have no shell, so
ENTRYPOINT ["myapp"]works and shell-formENTRYPOINT "myapp"does not. The static, base and cc images default to an empty entrypoint, soCMDmust also be exec form. - Run as non-root. Use the
nonroottags of distroless images, the non-root default of hardened images, or an explicitUSER. Pair this withrunAsNonRoot: trueandreadOnlyRootFilesystem: truein Kubernetes. - Pin the distribution and the digest. The distroless README advises referencing the Debian suffix explicitly because an unsuffixed tag will move to a newer Debian. Docker's build best practices recommend pinning to a digest (
FROM image:tag@sha256:...) for supply chain integrity and using automation to update digests on a schedule. - Test in a pilot service. Run functional and load tests, check TLS (CA bundle present), time zone handling, locale, and DNS resolution, and confirm health checks do not call a shell (for example a
HEALTHCHECK CMD curlwill fail without curl). - Roll out by service tier and keep the previous image digest available so a deployment can be rolled back by digest without rebuilding.
Building your own minimal images
Pulling a public minimal image is the fastest route, but larger estates often need their own internal bases with corporate CA certificates, specific runtime versions or approved libraries. Two declarative approaches avoid hand-written Dockerfiles that slowly regrow a full userland:
- apko with Wolfi packages: an apko YAML file lists repositories, packages, users and the entrypoint, and apko assembles the image directly from apk packages with no shell steps. The project describes builds as fully reproducible by default and generates an SBOM for each image, which simplifies proving that two builds are the same and that the SBOM matches the bytes.
- Bazel with rules_oci and rules_distroless: the distroless project itself is built with Bazel, and points to
rules_ocifor container images andrules_distrolessfor adding Debian packages, CA certificates and non-root users. This suits organisations that already use Bazel for application builds. - Dockerfile from scratch: workable for static binaries, but keep the file small: copy the binary, certificates and a passwd entry, set
USER, and nothing else.
| Runtime | Typical minimal base | Watch for |
|---|---|---|
| Go | distroless/static or scratch | Build with CGO_ENABLED=0 or the binary will need a libc; include CA certificates for outbound TLS |
| Java | distroless/java21-debian13 or a hardened JRE image | Distroless supports only the current Debian LTS Java; JVM options must go in exec-form arguments or environment variables |
| Node.js | distroless/nodejs24-debian13 | Only even-numbered current, active or LTS releases are supported; native add-ons must be built against glibc |
| Python | distroless/python3-debian13 or a hardened Python image | Copy a prepared virtual environment or site-packages from the builder; wheels built on musl will not load on glibc and vice versa |
| Rust and C or C++ | distroless/cc | Confirm the runtime library set covers libgcc and any linked libraries; ldd is not present, so check linkage in the builder |
Whichever route you choose, sign internal bases in your own pipeline and publish their SBOMs next to them, so downstream teams can verify a base the same way they verify a public one.
Debugging images without a shell
- Debug tags: each distroless image set has a
:debug(ordebug-nonroot) variant with a busybox shell. Use it in development, never as the production tag. - Ephemeral containers: the Kubernetes documentation describes
kubectl debugadding an ephemeral container to a running Pod, and names distroless images as a case wherekubectl execis insufficient. The--targetflag places the debug container in the target container's process namespace, and it must be supported by the container runtime. - Debug a copy:
kubectl debug --copy-tocreates a copy of the Pod with a different image, which avoids touching the production Pod. - Control debugging: ephemeral containers are a powerful capability. Restrict the
pods/ephemeralcontainerssubresource in RBAC to on-call roles and alert on its use. See the Kubernetes RBAC Guide.
Keeping minimal images patched and verified
- Rebuild often: Docker's best practices advise rebuilding images regularly with updated dependencies and using
--pullto fetch the latest base. Distroless tracks Debian and opens automated pull requests when Debian packages update. - Know the support window: the distroless support policy sets end of life for Debian 13 images at Debian 14 release plus one year for static, base and cc, and plus three months for Java, Node.js and Python images. Track this as you would any operating system lifecycle.
- Verify signatures: the distroless README states all images are signed by cosign with keyless signatures, verifiable against the issuer
https://accounts.google.comand the identity[email protected]. Enforce verification for every base in CI or at admission. See the Kubernetes Admission Control Guide. - Use SBOM and VEX: ingest the image SBOM rather than relying solely on scanner package detection, which can miss packages in images without a package database, and use vendor VEX statements to suppress non-exploitable findings with evidence. See the SBOM Guide and the VEX Guide.
- Check vulnerability data sources: make sure your scanner supports the distribution's advisory feed (Debian, Alpine secdb, Wolfi advisories) or OSV, otherwise findings will be missing or wrong.
Where identity fits
Minimal images remove tools an attacker would use to harvest credentials, but they do not remove the credentials. A distroless container still receives a mounted service account token, cloud workload identity and environment variables. Keep secrets out of image layers (the builder stage is a common leak point when files are copied wholesale), set automountServiceAccountToken: false where the workload does not call the Kubernetes API, and treat registry pull and push credentials as privileged non-human identities. See the Secrets in Container Images and Build Artefacts Guide and the Docker Hub Tokens Security Guide.
Detection and measurement
Minimal images change what you can detect inside the container: no shell means any shell process in a production container is a strong signal. Monitor:
- Process execution of
sh,bashorbusyboxin containers built from shell-less bases (runtime security tools or audit rules). - Deployments referencing tags rather than digests, or bases outside the approved list.
- Use of
kubectl debugand ephemeral containers in the Kubernetes audit log. - Images failing signature verification at admission.
| Metric | How to measure | Direction |
|---|---|---|
| Share of production workloads on approved minimal bases | Registry or cluster inventory | Up |
| Median OS-package findings per image | Scanner results grouped by base | Down |
| Base image age (days since base digest was published) | Compare deployed digest to latest | Down, for example under 30 days |
| Images deployed by digest with verified signatures | Admission controller logs | Up, towards 100% |
| Workloads running as root | Pod security reports | Down |
Common failure patterns
- Shipping the builder: a single-stage Dockerfile on a "minimal" base that then installs compilers and package caches defeats the purpose.
- Using debug tags in production: a
:debugimage reintroduces a shell, often because someone needed it once during an incident. - Pinning forever: digest pinning without automated updates freezes CVEs in place. Pin and update, never just pin.
- Mixing ecosystems: adding Alpine packages to a Wolfi image, or copying glibc binaries into Alpine. The Wolfi project says mixing packages with other distributions is unsupported and can create security problems.
- Trusting scanner silence: a scanner that cannot identify the distribution may report zero findings. Validate scanner coverage on each base before celebrating.
- Breaking health checks and entrypoints: shell-form commands and curl-based health checks fail silently in testing environments that still use the old base.
Framework mapping
| Practice | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | MITRE ATT&CK risk reduced |
|---|---|---|---|
| Remove unneeded packages and shells | CM-7 Least Functionality | PR.PS-01 | T1610 Deploy Container (less post-compromise tooling) |
| Approved base image baseline | CM-2, CM-6 | PR.PS-01 | T1525 Implant Internal Image |
| Rebuild and patch on a schedule; track end of life | SA-22, RA-5 | PR.PS-02, ID.AM-08 | Not technique specific |
| Verify signatures and provenance | CM-14, SR-4, SI-7 | PR.PS-01 | T1195.002 Compromise Software Supply Chain |
| Non-root, read-only runtime | AC-6 | PR.PS-01 | T1611 Escape to Host |
Practitioner checklist
- Inventory every base image and record finding counts before migration.
- Adopt multi-stage builds so only artefacts reach the runtime image.
- Choose the base by libc and runtime: distroless or Wolfi for glibc, Alpine only where musl is tested.
- Use exec-form entrypoints and non-root users; set
runAsNonRootand a read-only root filesystem. - Pin bases by digest and automate digest updates and rebuilds.
- Verify base image signatures in CI and at admission.
- Restrict and audit ephemeral container use; ban
:debugtags in production. - Track base image end of life and scanner coverage for each distribution.
Standards and references
- Google: Distroless container images
- Google: Distroless support policy
- Wolfi: Wolfi OS project overview
- Chainguard: apko image builder
- Alpine Linux: About
- Docker: Docker Hardened Images documentation
- Docker: Building best practices
- Kubernetes: Debug Running Pods
Related NHI Mgmt Group resources: Container Image Security Guide · NIST SP 800-190 Container Security Guide · n8n Security Guide · Vulnerability Prioritisation Guide