Terraform is not a recovery plan for cloud infrastructure incidents

At a glance

What this is: This is an analysis of why Terraform and recovery tooling solve different cloud operations problems, with the key finding that Terraform state is not a substitute for restoring actual deployed infrastructure.

Why it matters: It matters to IAM and security teams because infrastructure recovery increasingly depends on identity, access, and configuration fidelity, especially when out-of-band changes, drift, or emergency fixes occur outside normal governance.

👉 Read Commvault's analysis of Terraform recovery limits and actual-state rewind

Context

Infrastructure-as-code makes provisioning repeatable, but repeatability is not the same as recoverability. Once teams rely on Terraform state and Git history as a recovery path, they create a governance gap: they can redeploy declared configuration, but not necessarily the exact runtime environment that existed before an incident. In cloud operations, that distinction becomes critical when manual console changes, drift, or partial failures are involved.

The identity angle is indirect but real. Recovery in cloud environments often depends on access paths, service identities, and configuration changes that were introduced outside the normal IaC pipeline. When those changes are not represented in code, organisations can lose the ability to prove what changed, who changed it, and how to return to a known-good state. That makes cloud recovery part of broader control assurance, not just an operations concern.

Key questions

Q: What breaks when Terraform is treated as a recovery tool?

A: Teams can redeploy desired configuration, but they may not be able to reconstruct the exact environment that was running before the incident. That becomes a serious problem when console edits, manual fixes, or drift changed production outside the code path. Recovery then depends on incomplete records instead of actual state.

Q: Why do cloud teams need both IaC and recovery tooling?

A: IaC makes provisioning repeatable, while recovery tooling preserves what was actually deployed. Those are different control objectives. Without both, teams can build quickly but still fail to restore accurately after deletion, drift, or a bad deployment.

Q: How do organisations know if their rollback process is actually working?

A: A real test is whether the team can restore a known-good environment after a change made outside Terraform. If the process only works when code, state, and history are complete, then it is not a reliable recovery capability.

Q: What is the difference between desired state and actual state in cloud operations?

A: Desired state is the configuration infrastructure-as-code says should exist. Actual state is the environment that is truly running at a given moment. Security and resilience teams need both views because incidents, drift, and manual intervention often affect actual state first.

Technical breakdown

Desired state vs actual deployed state

Terraform manages desired state. It records what infrastructure should look like and then reconciles the environment toward that configuration. That is powerful for provisioning, but it is not the same as capturing runtime reality. If a resource is changed in the console, modified by a script, or drifts over time, the Terraform plan may no longer describe what was actually running. Recovery from desired state can recreate a configuration, but it cannot guarantee restoration of the precise deployed condition that existed before the incident.

Practical implication: separate provisioning controls from recovery controls so a successful Terraform apply is never mistaken for a point-in-time restore.

Why Terraform state and Git history are not recovery artefacts

Terraform state files describe managed infrastructure relationships, and Git history records code changes. Neither one is a full snapshot of operational reality. They may omit manual edits, emergency changes, deleted resources, transient settings, and environment drift. In incident conditions, that gap matters because the environment you need to restore may not match the last committed configuration. Recovery artefacts need to preserve actual deployed state, not just the intended or previously declared state.

Practical implication: treat state files and repository history as change-management evidence, not as your authoritative rollback mechanism.

Point-in-time recovery depends on captured cloud configuration

Point-in-time recovery works by preserving the actual deployed environment at a specific moment, then rewinding to that version when a deployment fails or resources disappear. That model is fundamentally backward-looking. It supports restoration after accidental deletion, bad changes, or out-of-band modifications because it reconstructs what was live, not what code said should exist. The operational difference is subtle but decisive: build tools define the target, while recovery tools preserve the evidence needed to return there.

Practical implication: maintain a recoverability layer that snapshots live cloud configuration independently of IaC pipelines.

Threat narrative

Attacker objective: The operational objective is not compromise but interruption, with the practical effect of forcing teams to recover from incomplete source-of-truth records.

1. Entry occurs through a failed deployment, accidental deletion, or manual out-of-band modification that changes production infrastructure outside normal IaC workflows.
2. Escalation happens when teams discover that Terraform state and Git history cannot reconstruct the actual running environment with enough fidelity to reverse the damage.
3. Impact is extended downtime, configuration uncertainty, and delayed restoration because the environment must be rebuilt instead of rewound.

NHI Mgmt Group analysis

Terraform drift has become a recoverability problem, not just a configuration problem. Once teams accept that infrastructure can change through consoles, scripts, and emergency fixes, they also accept that code history will not always tell the full story. That creates a control gap between change management and incident recovery. The practical conclusion is that cloud governance must distinguish declared state from actual state.

Recovery assurance is now part of identity and access governance in cloud operations. In practice, many of the changes that create drift are performed through privileged access paths, service accounts, or ad hoc administrative credentials. That means the ability to restore an environment depends on how well those identities were governed at the moment the change occurred. Strong IAM does not replace recovery tooling, but it reduces the number of unknown changes that recovery has to absorb.

Point-in-time restore is a separate control objective from infrastructure provisioning. Terraform can rebuild a configuration, but it cannot prove that the rebuilt environment matches the one that was actually live before disruption. That is why the market should stop treating IaC and recovery as interchangeable categories. The practitioner lesson is to evaluate provisioning, state management, and live-state capture as distinct layers.

Actual-state capture is the named concept this discussion sharpens. The important governance question is not whether a tool can recreate code, but whether it preserves the environment that was truly running when the incident began. That concept matters across cloud security, resilience, and identity governance because incident response fails when source-of-truth systems omit the operational facts needed to recover. Practitioners should test for actual-state capture, not just config replay.

Cloud resilience now depends on preserving the evidence of change. If teams cannot reconstruct who altered the environment, what changed, and which version was live, then they cannot restore with confidence. That makes recovery telemetry and state capture part of the control fabric, not an optional enhancement. Practitioners should align recovery design with auditability and operational assurance.

What this signals

Cloud resilience programmes should now treat actual-state capture as a control requirement, not a convenience feature. If recovery cannot reconstruct the environment that was truly live, then change management metrics may look healthy while incident restoration remains fragile. The operational lesson is to test whether your restore path works after manual changes, failed deployments, and drift.

State fidelity gap: this is the governance gap between what code says exists and what the cloud was actually running. That gap widens when privileged access, emergency intervention, or unmanaged scripts sit outside the IaC pipeline. Teams that care about recovery assurance should monitor drift as a resilience signal, not just a configuration hygiene issue.

For practitioners

• Separate provisioning from recovery controls Use Terraform for desired-state management and a distinct recovery mechanism for live-state capture and point-in-time restore. Do not rely on Git history or Terraform state as your rollback path when production changes fail or drift outside the pipeline.
• Identify drift-prone change paths Map where console edits, scripts, break-glass accounts, and service identities can change cloud infrastructure outside IaC. Prioritise those paths for monitoring because they are the places where actual deployed state diverges from declared state.
• Test restoration against real incidents Run recovery exercises for deleted resources, failed deployments, and out-of-band modifications. The test should confirm that your team can restore the environment that was actually running, not merely reapply the last approved configuration.
• Preserve live-state snapshots separately Create a recoverability layer that captures deployed cloud configuration at specific points in time and keep it independent from source control. This gives incident responders a reliable rewind point when code and state files are incomplete.

Key takeaways

• Terraform is a provisioning control, not a recovery control, so its state file cannot stand in for actual runtime restoration.
• The operational risk is environment drift, because manual changes and console actions can leave code history unable to reconstruct what was live.
• Practitioners should design for actual-state capture and point-in-time restore as separate capabilities from infrastructure provisioning.

Key terms

• Desired state: The target configuration or operating condition an administrator wants a device to reach and maintain. In declarative systems, desired state replaces many direct commands with policy, rules, and configuration data. The practical challenge is ensuring the declaration is accurate, complete, and auditable across the fleet.
• Actual State: The cloud environment that is truly running at a specific point in time. Actual state includes manual edits, drift, emergency changes, and runtime differences that code repositories and state files may not fully capture.
• Configuration Drift: Configuration drift is the gradual divergence between a system's intended secure state and the settings it actually runs with over time. In SaaS, drift often appears when admins change sharing, logging, or access controls under pressure and never return to validate the result.
• Point-in-Time Recovery: Point-in-time recovery restores data or assets to the exact state they had at a specific moment before deletion, corruption, or unwanted change. In SaaS analytics environments, it must preserve the object and the dependencies needed for it to function, not only a static export.

What's in the full article

Commvault's full article covers the operational detail this post intentionally leaves for the source:

• How the vendor distinguishes desired-state rebuilds from actual-state rewind in practical cloud recovery workflows
• Examples of incident scenarios where Terraform state and Git history fail to restore the running environment
• The specific recovery use cases the source maps to point-in-time restoration after deletion, drift, or failed deployments
• The vendor's explanation of how Cloud Rewind fits alongside existing infrastructure-as-code processes

👉 Commvault's full article covers point-in-time restore scenarios, drift recovery, and how the two tools fit together

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader recovery and resilience decisions that cloud programmes depend on.