Why should Kubernetes containers use a read-only root filesystem?

Why a read-only root filesystem matters in Kubernetes

A read-only root filesystem turns the container’s root layer into immutable application code rather than a writable workspace. That is valuable because containers are meant to be disposable, but attackers often rely on dropping binaries, modifying startup files, or altering local state to survive longer than the original compromise. The setting makes those actions harder and narrows the room for unexpected change.

It also helps you separate what the application needs to write from what it should never touch. If a workload truly needs scratch space, logs, uploads, or cache, those should be mounted as explicit writable volumes instead of being hidden inside the image layer. That distinction improves predictability and makes the container’s runtime behaviour easier to reason about.

For Kubernetes operators, this is not just a hardening preference. A read-only root filesystem is a practical control that complements immutable images, pod security settings, and runtime monitoring. NHIMG’s Kubernetes Security Guide places this control in the broader context of cluster hardening, including runtime and admission controls that help keep the deployment model consistent.

What changes when the root layer is writable

When the root filesystem is writable, a compromised process has more options after initial execution. It may try to plant a helper binary, edit scripts, tamper with application configuration, or leave behind files that support repeated access. Even if those changes are small, they can complicate incident response because the container no longer reflects the exact artifact that was originally deployed.

The risk is not limited to deliberate attacker activity. Writable root access also makes accidental drift more likely, because misbehaving code, debugging shortcuts, or temporary fixes can persist inside a running container. Over time, that can produce snowflake containers whose runtime state diverges from the image you built and tested. For container image hygiene and the wider integrity problem, see Container Image Security Guide, which covers the image-layer issues that a read-only root filesystem helps contain.

That control is especially relevant when application code, package managers, shell tools, or scripting runtimes would otherwise be able to write into paths under /. The less the container can change itself, the easier it is to detect abnormal behaviour and the less likely a simple compromise becomes a durable foothold.

Writable root access also matters when secrets or credentials are embedded poorly in images or runtime files. A container that can write to its own filesystem can often stage those materials for reuse or exfiltration, which is why the image-layer concern often overlaps with credential exposure. NHIMG’s Massive Docker Hub Secrets Leak and Secrets in Docker Hub images (RWTH Aachen study) both show why image contents and filesystem writability should be treated as part of the same hardening problem.

How to make it work without breaking the application

The main design rule is simple: the image root should stay immutable, and only the directories that genuinely need runtime writes should be mounted separately. Common examples include temporary working directories, application cache, upload paths, and log destinations. If the workload fails when root is read-only, that usually means the application is writing somewhere it should not, or it was packaged with an unnecessary dependency on local mutation.

A practical way to implement this is to test the container in a restrictive mode early, then enumerate each write path the application still needs. The goal is not to preserve convenience for the process, but to force deliberate choices about where state lives. If a path is needed only during startup or debugging, treat that as a packaging issue rather than a runtime requirement.

One useful discipline is to pair the read-only root filesystem with minimal images and explicit volume mounts. That combination reduces the number of places an attacker can modify and reduces the number of places your own team has to inspect when something looks wrong. If you need a reference point for the surrounding runtime and image-risk controls, NIST SP 800-190 Container Security directly addresses image, registry, orchestrator and runtime risk.

Another useful check is whether the workload still behaves correctly when unexpected writes are blocked. If it cannot, that is often a sign that the application has hidden dependencies on mutable state, rather than a sign that the control is too strict. In those cases, the fix is usually to redesign the write paths, not to weaken the filesystem boundary.

Risk and Threat Considerations

A writable root filesystem increases the blast radius of a container compromise because an attacker or malicious payload can try to persist inside the container, alter files that affect execution, or stage tools for later use. It also weakens forensic confidence, because the running container may no longer match the image that was originally deployed.

Failure mechanism: Runtime code gets write access to the image layer, then uses that access to plant files, alter application behaviour, or create persistence that survives longer than the original process session.

Impact: You get more durable compromise, noisier incident response, and a larger gap between the trusted artifact and the actual container state.

Practitioner Guidance

What to verify: Confirm that every write the application performs is intentional, mounted on a separate volume, and needed for normal operation. If the container writes to the root filesystem during steady state, treat that as a packaging or design defect to fix rather than a control to waive.

Common mistake: Teams often enable a read-only root filesystem only after discovering a failure in production, then soften the control until the application works. A better approach is to discover the write dependencies in test, remove unnecessary writes, and keep the runtime policy strict.

Practitioner takeaway: The control works best when it is paired with immutable images and explicit state separation, because the real objective is not just blocking writes, but preventing hidden runtime drift from becoming a persistence mechanism.

Related resources from NHI Mgmt Group

• Why should Amazon ECS containers use read-only root filesystems?
• When does regex-based secret detection become too unreliable for production use?
• What breaks when pods run as root or privileged containers are allowed in Kubernetes?
• What is the difference between running containers as root and using explicit privilege controls in Kubernetes?