Immutable Infrastructure Security Guide: Replace-Not-Patch Operations, No-SSH Hosts and Rebuild Cadence

Servers that are logged into, patched and tweaked in place drift away from any known state, and every interactive login is both an audit gap and a route for persistence. Immutable infrastructure answers this by never changing a running host: a new image is built, tested and deployed, and the old host is destroyed. Done properly, it removes SSH keys from production, puts a ceiling on how long an attacker's foothold can last and turns patching into a deployment. Done partially, it leaves a fleet of "immutable" hosts with open SSH, mutable state on the root disk and secrets that survive every rebuild. This guide covers the operating model rather than the image build: what must be true for a host to be immutable, removing interactive access and handling emergencies, rebuild cadence and replacement mechanics, state and secrets, and incident response when the evidence is designed to disappear. It is for platform, SRE and cloud security teams. Image pipelines are covered in the Server Build and Golden Image Hardening Guide.

Key takeaways

• Immutable means no in-place change at all: AWS Well-Architected REL08-BP04 defines it as no updates, security patches or configuration changes in place on production workloads; anything less is a mutable estate with good intentions.
• Remove SSH, then design the exceptions: use an audited, identity-based session service for emergencies, disabled by default and alerting when used.
• Set a maximum host age: a rebuild cadence (for example 14 to 30 days) limits how long any compromise or missed patch can persist, and cloud autoscaling can enforce it.
• Replacement does not rotate secrets: credentials a host could read must be rotated separately after a compromise, or the attacker keeps access through the next generation.
• Capture evidence before you destroy: replace-on-suspicion is a strong containment step, but snapshot disks and memory first.

What makes a host immutable

AWS Well-Architected's reliability pillar describes the model directly: when a change is needed, new infrastructure is built and deployed into production, deployments are validated automatically, and traffic shifts gradually to the new resources. Its listed benefits include reduced configuration drift, atomic deployments, fast rollback to the unchanged previous version and an improved security posture because "remote access mechanisms (such as SSH) can be disabled". In practice a host is immutable when five conditions hold.

Purpose-built operating systems make several of these properties default. Bottlerocket's security features list a read-only root filesystem backed by dm-verity, a stateless tmpfs for /etc, no shell or interpreters, SELinux enforcing, kernel lockdown in integrity mode, Secure Boot, and updates applied by flipping between two partition sets. Google's Container-Optimized OS mounts the root filesystem read-only, verifies its checksum on each boot and keeps /etc writable but stateless. Talos Linux states that all system management is done through an API secured with mutual TLS, with no shell or interactive console. General-purpose distributions can approach the same model with read-only root mounts, removal of SSH and strict pipeline control, but need more work to prove it.

No-SSH hosts and emergency access

SSH on an immutable host is a contradiction: it exists to change things, and changes made through it are lost or, worse, persist where they should not. Removing it also removes a large set of non-human credentials (host keys, user keys, CI keys in authorized_keys) and a common persistence technique, ATT&CK T1098.004 SSH Authorized Keys. See the SSH Key Management Guide.

• Logging caveat: AWS notes that Session Manager logging is not available for sessions that connect through port forwarding or SSH. If you tunnel SSH over Session Manager, you lose the session transcript; prefer native shell sessions.
• Treat any session as a taint: a host that someone has logged into is no longer known-good. Tag it automatically when a session starts and replace it after the investigation ends.
• Restrict who can enable access: the permission to start a session or enable an admin container is privileged access. Grant it just in time, through approval, to named people. See the AWS Systems Manager Security Guide and the Break-Glass Account Guide.

Rebuild cadence and replacement mechanics

Immutable hosts are patched by replacement, so the rebuild cadence is the patch cadence. Two triggers drive it: a scheduled maximum age, and an emergency rebuild when a critical vulnerability or a known-exploited CVE lands in the image.

• Maximum host age: Amazon EC2 Auto Scaling's maximum instance lifetime terminates and replaces instances after a set time, with a minimum value of 86,400 seconds (one day). AWS names the use case directly: replacing instances on a schedule because of internal security policies or external compliance controls. Instances with scale-in protection are not replaced, so audit protection flags. Azure Virtual Machine Scale Sets and Google Cloud managed instance groups offer comparable automatic image upgrade and rolling replacement features.
• Rolling replacement: EC2 Auto Scaling instance refresh replaces instances in a rolling fashion after a new launch template version points at a new image, with checkpoints, skip matching and rollback available. Kubernetes node pools use node image upgrades with cordon and drain. See the Patch rings and update orchestration guide.
• Deployment patterns: Well-Architected recommends canary and blue/green deployments, with the previous version unchanged so that rollback is a traffic shift rather than a repair.

A rebuild only helps if the new image is actually newer. The image pipeline must rebuild from patched sources on its own schedule; replacing hosts with a three-month-old image just resets uptime. Track the age of the image, not only the age of the instance.

State, configuration and secrets

• Separate state: databases, queues and file data live on managed services or attached volumes that survive replacement. A root disk that holds data forces in-place patching. See the Kubernetes Storage Security Guide for the container equivalent.
• Configuration through code: launch templates, user data and Kubernetes manifests are the only way to change a host, so the infrastructure as code pipeline and its credentials become the control point. Protect them as production access. See the Infrastructure as Code Security Guide.
• Secrets at runtime, never in images: hosts obtain credentials from the platform's workload identity or a secrets manager at boot, so a new generation of hosts does not need anyone to copy keys.
• Rotation is separate from replacement: terminating a compromised host does not revoke anything it could read. The ToolShell SharePoint exploitation showed attackers keeping access after patching because stolen machine keys stayed valid. After a suspected compromise, rotate every secret, token and key the host's identity could reach, then replace. See the Secrets Rotation Runbook Template.

Incident response on disposable hosts

Replacement is an effective containment and eradication step, because persistence on the root disk (T1543 Create or Modify System Process, T1505 Server Software Component) disappears with the host. It also destroys evidence. Build the forensic steps into the automation:

1. Isolate, do not terminate: move the suspect instance to a quarantine security group, detach it from load balancers and set termination protection.
2. Capture: snapshot volumes and capture memory where the platform allows, and copy logs that were not yet shipped.
3. Rotate: revoke the instance role sessions and rotate any secret it could read.
4. Replace: launch a clean generation from a known-good image and verify the image itself was not the source (ATT&CK T1525 Implant Internal Image).
5. Review: compare the captured disk with the image to find what changed. In an immutable estate, any difference is a finding.

See the Cloud Forensics Guide and the Cloud Incident Response Guide.

Implementation: a phased rollout

1. Pick one stateless tier (a web or worker service behind a load balancer) and confirm it holds no state on its root disk.
2. Automate the image with a pipeline that rebuilds on a schedule and on critical fixes, and publishes an approved image ID.
3. Ship logs and metrics off-host so that diagnosis does not need a login.
4. Introduce the emergency path (session service or disabled-by-default admin container) and test it before removing SSH.
5. Remove SSH in audit mode: close the security group rule but leave the daemon for one cycle, and watch for anyone who needs it. Then remove the daemon and keys from the image.
6. Enforce a maximum age and rolling replacement, starting with a long lifetime and shortening it as confidence grows.
7. Block in-place change: deny IAM actions that modify running instances outside the pipeline, and alert on sessions.
8. Roll back by pointing the launch template or node pool at the previous image; keep at least two previous approved images available.

Small teams can get most of the value with a managed container platform and a container-optimised OS, where the provider handles node images. Large estates need the full pipeline, policy enforcement and metrics, and will keep a mutable tier for some legacy workloads; manage that tier with the Configuration Drift and Baseline Enforcement Guide.

Detection and measurement

• Alert on: any interactive session start; security group changes opening port 22 or 3389; instances launched from non-approved images; changes to launch templates outside the pipeline; disabled scale-in protection; drift reported by IaC drift detection.
• Host signals: on read-only OSes, integrity failures at boot are high-severity events. On general-purpose OSes, file integrity monitoring of system paths should be silent.

Common failure patterns

• Immutable in name only: hosts are built from images, then patched by an agent and accessed by SSH, so drift returns and the model's guarantees vanish.
• Stale images: hosts are replaced weekly from an image built months ago.
• Break-glass that became the norm: the emergency session path is used daily and never followed by replacement.
• Termination as the only response: a suspect host is destroyed by autoscaling before anyone captures disk or memory.
• Secrets outlive hosts: long-lived keys in user data, environment variables or the image survive every generation and every compromise.
• Pipeline left unprotected: attackers who cannot reach a host target the image pipeline or IaC repository instead. See the IaC platform compromise playbook.

Framework mapping

Relevant ATT&CK techniques: T1098.004 SSH Authorized Keys, T1021.004 SSH, T1543 Create or Modify System Process, T1505 Server Software Component and T1525 Implant Internal Image.

Practitioner checklist

• Define which tiers are immutable and confirm none hold state on root disks.
• Rebuild images on a schedule and on critical or KEV-listed fixes, and track image age.
• Set and enforce a maximum host age per tier, and audit scale-in protection.
• Remove SSH daemons, keys and inbound management rules from immutable images.
• Provide one audited, identity-based emergency access path, granted just in time.
• Tag any host that receives an interactive session and replace it afterwards.
• Deny in-place modification of running instances outside the pipeline.
• Automate isolate, capture, rotate and replace for suspect hosts.
• Protect the image pipeline and IaC repositories as production access.

Standards and references

• AWS Well-Architected: REL08-BP04 Deploy using immutable infrastructure
• AWS: Replace Auto Scaling instances based on maximum instance lifetime
• AWS: AWS Systems Manager Session Manager
• Bottlerocket: Security Features
• Google Cloud: Container-Optimized OS security overview
• Sidero Labs: Talos Linux
• NIST: SP 800-53 Rev. 5
• MITRE ATT&CK: SSH Authorized Keys

Related NHI Mgmt Group resources: Server Build and Golden Image Hardening Guide · Configuration Drift and Baseline Enforcement Guide · SSH Key Management Guide · Patch Management Guide