Kubernetes Backup and Recovery: Back Up Clusters and Workloads You Can Actually Restore

Most Kubernetes backups fail at restore time, not backup time, because nobody proved that the objects, the volume data, the secrets and the keys that decrypt them come back together in a working cluster. A Kubernetes cluster holds state in three places: etcd (every API object), persistent volumes (application data) and whatever sits outside the cluster that the cluster depends on (Git repositories, container registries, cloud identities, key management). A backup plan that covers only one of them gives you a partial restore at best. This guide covers etcd snapshots, Velero object and volume backups, CSI volume snapshots, GitOps rebuilds, secret and key handling, and restore testing. It is written for platform engineers, SREs and security teams running self-managed or managed Kubernetes. General backup principles (3-2-1-1-0, immutability, isolated vaults) are covered in the Backup Security and Immutable Backup Guide; this page applies them to Kubernetes.

Key takeaways

• Decide what you are protecting against first. Losing a control plane, losing a namespace, losing a storage system and a ransomware actor holding cluster-admin each need a different backup layer and a different restore drill.
• etcd snapshots restore the whole cluster or nothing. They are the right tool for control plane disaster on self-managed clusters, the wrong tool for recovering one deleted Deployment, and unavailable on managed services where the provider runs etcd.
• Velero restores objects, not guarantees. It skips resources that already exist, needs the same API group versions on the target and, for file system backups, uses a common repository key, so access to the backup bucket is access to the data.
• GitOps rebuilds configuration, not state. Git can recreate Deployments and policies, but not database contents, generated secrets, certificates or anything changed by hand outside the pipeline.
• A backup is only proven by a timed restore into a clean cluster. Measure restore time and restored data correctness, not backup job success.

What needs backing up in a Kubernetes cluster

Treat a cluster as layers, each with its own owner and recovery method. The table below is the inventory most teams build before choosing tools.

Managed control planes change the first row. On Amazon EKS, Azure Kubernetes Service and Google Kubernetes Engine the provider operates etcd and, in the standard managed offerings, does not give customers a way to take or restore their own etcd snapshot, so object-level backup (Velero or the provider's own backup service) plus volume snapshots and GitOps carry the whole burden. Each provider also has its own disaster recovery documentation for its managed backup service; read it alongside this page.

etcd snapshots for self-managed clusters

The Kubernetes documentation is direct: all Kubernetes objects are stored in etcd, periodic etcd backup is needed to recover from disasters such as losing all control plane nodes, and because the snapshot file contains all Kubernetes state, the snapshot files should be encrypted. Two methods are supported: the etcd built-in snapshot (etcdctl snapshot save against a live member, which the documentation says does not affect the member's performance) and a snapshot of the storage volume etcd runs on.

• Prefer the built-in snapshot: the etcd recovery guide notes that a snapshot taken with etcdctl snapshot save carries an integrity hash that etcdutl snapshot restore checks, while copying the member/snap/db file can lose data that was still in the write-ahead log.
• Verify every snapshot: run etcdutl snapshot status and record the hash, revision, key count and size. A sudden drop in key count is a signal worth alerting on. The etcdctl snapshot status form is deprecated from etcd v3.5 in favour of etcdutl.
• Authenticate with client certificates: snapshots are taken over the TLS client port (2379) using the CA, certificate and key from the etcd Pod definition. Those certificates are effectively cluster-admin over storage; keep the snapshot job's credentials out of general reach.
• Restore the whole cluster from one file: the etcd guide states that all members should restore from the same snapshot, and etcdutl snapshot restore creates new data directories.
• Stop API servers first: the Kubernetes procedure is to stop all API server instances, restore state in all etcd instances, then restart the API servers, and it recommends restarting kube-scheduler, kube-controller-manager and kubelet so they do not rely on stale data.
• Use revision bumps: clients and controllers may see the etcd revision go backwards after a restore. The etcd guide recommends restoring with --bump-revision and, for Kubernetes, --mark-compacted so existing watches are terminated rather than served inconsistent history.
• Back up before upgrades: the Kubernetes documentation states to back up etcd before starting an etcd upgrade. Restores are supported from snapshots of the same major.minor etcd version.

If a kubeadm cluster is restored into a new data directory, update etcd.local.dataDir in the kubeadm-config ConfigMap as the documentation describes, or the next kubeadm upgrade will write a manifest pointing at the old directory.

Velero: object and volume backup

Velero is the open source tool most teams use for object-level backup. Each backup, schedule and restore is a custom resource; the backup controller queries the API server and uploads a tarball of objects to object storage, and by default also triggers disk snapshots of persistent volumes. Velero treats object storage as the source of truth and syncs backup records from the bucket into a new cluster, which is what makes cross-cluster migration and rebuild-then-restore work. Hardening the tool itself is covered in the Velero Security guide.

Behaviours that surprise people at restore time

• Backups are not atomic: the Velero documentation says objects created or edited during a backup might not be included. Quiesce or use backup hooks for applications that need consistency, for example flushing a database before its snapshot.
• Non-destructive by default: if a resource already exists in the target cluster, Velero skips it. ServiceAccounts are the exception (fields are merged). --existing-resource-policy=update patches existing resources on a best-effort basis but does not overwrite PV data; --existing-volume-data-policy handles data in bound PVCs.
• API versions must match: objects are backed up at the API server's preferred version for each group and resource, and that version must exist on the target. Restoring an old backup into an upgraded cluster can fail for removed beta APIs and CRD versions.
• Status is dropped: Velero removes the status field before restore unless you select resources for status restoration, which matters for custom resources that store state in status.
• Restore order is fixed: CRDs, Namespaces, StorageClasses, snapshot classes and contents, PVs and PVCs, then RBAC, ServiceAccounts, Secrets, ConfigMaps and finally workloads. Custom priorities via --restore-resource-priorities apply to all future restores, so change them deliberately.
• Retention deletes everything: the default TTL is 30 days, and expiry removes the backup record, the bucket objects, the volume snapshots and associated restores. Set retention from your recovery point requirements, not the default.

Choosing a volume method

For most production data the order of preference is: application-consistent snapshot with the data moved off the source storage, then plain CSI snapshot, then file system backup where nothing else fits. The Kubernetes snapshot API's deletionPolicy decides whether deleting a VolumeSnapshot deletes the underlying storage snapshot; note that Velero patches its own snapshot contents to Delete when the backup expires, regardless of the class setting.

Protecting backups from the cluster that made them

A ransomware actor with cluster-admin can delete Velero Backup objects, and if the Velero service identity can delete bucket objects, the attacker can use it too. The Codefinger S3 ransomware case showed the shape of the risk: stolen long-lived access keys with write access to buckets were enough to re-encrypt objects and set lifecycle rules to delete them. A Velero bucket written with a static key from inside the cluster is exactly that exposure.

• Separate the write path from the delete path: give the backup identity permission to put objects but not to delete them or change bucket retention, and enforce immutability (object lock or the provider equivalent) on the backup bucket.
• Use workload identity, not static keys: bind Velero to a cloud role through the cluster's workload identity mechanism rather than a cloud access key in a Kubernetes Secret. See the Kubernetes NHI Security Guide.
• Put backups in a different account and region: a backup storage location in the same cloud account as the cluster shares its blast radius. Use a separate account or project with its own administrators, as the Backup Identity Isolation Guide describes.
• Restrict who can create Restore objects: a restore can recreate privileged RBAC bindings and Secrets. Treat create rights on Velero CRDs as privileged, alongside the controls in the Kubernetes RBAC Guide.
• Set the backup location read-only during an incident restore: Velero supports read-only backup storage locations, which stops backups being created or deleted while you restore.

Secrets and keys: what makes a restore readable

Secrets are where Kubernetes backups most often turn into a security incident or a dead restore. Three facts drive the design.

• Backups contain plaintext-equivalent secrets: etcd snapshots hold every Secret, and Velero backs up Secret objects as API objects. Anyone who can read the bucket or the snapshot file can read the cluster's credentials. Encrypt at rest with keys the backup operator cannot casually use, and scope bucket read access as tightly as production secret access.
• Encryption at rest keys must survive the cluster: if the API server encrypts Secrets in etcd using an EncryptionConfiguration or a KMS provider, an etcd snapshot is useless without those keys. The Kubernetes documentation warns that losing all copies of an encryption key means deleting the resources encrypted under it. Back the keys up separately, under different control, and test that a restored cluster can decrypt.
• Controller keys for sealed or external secrets: a GitOps repository full of sealed secrets cannot be decrypted by a new controller with a new key pair. Back up the controller's private key under break-glass control, or design so that secrets are pulled from an external secrets manager the rebuilt cluster can reach. See SOPS and Sealed Secrets and the Kubernetes Secrets Security Guide.

After a compromise, do not restore secrets as they were. Restore the workloads, then rotate every credential that was in the cluster, because the backup also captured whatever the attacker may have read. Service account token rotation is covered in How to rotate a Kubernetes service account token.

GitOps rebuild versus restore

A GitOps controller reconciling from Git turns cluster recovery into "build a new cluster and point it at the repository". That is the cleanest answer after a compromise, because you avoid restoring objects an attacker may have planted. It is not a full answer.

To make the rebuild path real, keep the Git repository, the container registry and the secrets source available independently of the cluster and region you are recovering. Anything applied with kubectl by hand is drift and will be missing. The GitOps Security Guide covers protecting the repository and controller.

Implementation: a phased approach

1. Classify workloads: list every namespace with an owner, a recovery point objective (RPO), a recovery time objective (RTO) and whether it is stateless (rebuild from Git) or stateful (needs data backup).
2. Close the GitOps gap: find objects not in Git (compare live objects with repository output) and either bring them under GitOps or include them in Velero scope.
3. Stand up etcd snapshots on self-managed clusters: schedule etcdctl snapshot save at an interval matching the cluster's RPO, verify each snapshot with etcdutl snapshot status, encrypt it and ship it off the control plane nodes.
4. Deploy Velero with an isolated backup location: workload identity, write-only permissions, immutable bucket in a separate account and region, TTLs set per schedule, and CSI snapshots with data movement for stateful workloads.
5. Add application consistency: backup hooks or the database's native dump for systems that cannot be crash-consistent.
6. Back up keys separately: etcd encryption configuration or KMS key access, sealed secrets controller keys and the restore credentials themselves, held under break-glass control.
7. Test restores into a clean cluster: start with one non-critical namespace, then a full stateful application, then a full cluster rebuild. Record time to service and validate data, not only object counts.
8. Roll back safely: restore into a new namespace with namespace remapping or into a separate cluster first, compare, then cut traffic over. Never test with --existing-resource-policy=update against production.

Detection and measurement

• Alert on backup tampering: deletes of Velero Backup, Schedule or BackupStorageLocation objects in the Kubernetes audit log, bucket policy, lifecycle or retention changes, and object deletions in the backup bucket. See the Kubernetes Audit Log Detection Guide.
• Watch backup phases: alert on Backups in Failed or PartiallyFailed and on the velero.io/gc-failure label, which marks backups that failed to delete.
• Track snapshot health: etcd snapshot age, size and key count trend; a snapshot that is suddenly much smaller is a warning.

Framework mapping

Common failure patterns

• Backing up objects but not volumes: a Velero schedule with snapshots disabled, or a CSI driver whose snapshots live on the same storage, restores Deployments that start against empty databases.
• Static cloud keys with delete rights: the backup credential sits in a Secret in the cluster it protects and can delete the bucket contents, so one cluster compromise removes both production and backups, the pattern seen in the Codefinger S3 ransomware case.
• Losing the encryption keys: etcd snapshots or sealed secrets restored without the encryption configuration or controller key, leaving every Secret unreadable.
• Restoring into an upgraded cluster untested: removed API versions and changed CRDs make an old backup fail on the only day it is needed.
• Restoring the attacker's changes: a full restore after compromise brings back planted ServiceAccounts, RoleBindings, webhooks and CronJobs. Rebuild configuration from Git and inspect what you restore; see the Kubernetes Cluster Compromise Playbook.
• Treating "backup completed" as success: PartiallyFailed backups and expired TTLs go unnoticed until a restore drill, or a real incident, finds the gap.

Practitioner checklist

• Every namespace has an owner, RPO, RTO and a stated recovery method (rebuild, restore or both).
• Self-managed clusters take verified, encrypted etcd snapshots stored off the control plane, and the restore procedure (stop API servers, same snapshot on all members, revision bump) is written down.
• Velero uses workload identity, cannot delete or shorten retention in its bucket, and writes to an immutable location in a separate account and region.
• Stateful workloads use CSI or provider snapshots with data moved off the source storage, plus application hooks where crash consistency is not enough.
• Encryption at rest keys, KMS access and sealed secrets controller keys are backed up separately and tested for decryption in a restored cluster.
• Git, the container registry and the secrets source are available independently of the cluster and region being recovered.
• Audit log alerts exist for deletion of Velero objects and for backup bucket policy, lifecycle and object deletion events.
• A timed restore into a clean cluster runs at least quarterly, including one after each Kubernetes minor version upgrade.
• Post-compromise recovery includes rotating every credential that was present in the cluster.

Standards and references

• Kubernetes: Operating etcd clusters for Kubernetes
• etcd: Disaster recovery
• Kubernetes: Volume Snapshots
• Kubernetes: Encrypting Confidential Data at Rest
• Velero: How Velero Works
• Velero: Restore Reference
• Velero: Container Storage Interface Snapshot Support
• Velero: File System Backup

Related NHI Mgmt Group resources: Backup Security and Immutable Backup Guide · Kubernetes Security Guide · Cloud Native Backup and Ransomware Resilience Guide · Business Continuity and Disaster Recovery Guide