Cleanroom recovery for critical infrastructure: what SMMPA changed

At a glance

What this is: SMMPA’s recovery strategy centers on cleanroom restoration, air-gapped backups, and repeated validation for critical infrastructure resilience.

Why it matters: For IAM and NHI practitioners, this reinforces that recovery design must account for identity, access, and trust assumptions, because compromised credentials and restored services often fail together.

By the numbers:

• SMMPA serves 17 municipal utilities.
• SMMPA supports critical power infrastructure with approximately 50 employees.

👉 Read Commvault's case study on Cleanroom Recovery at SMMPA

Context

SMMPA’s story is really about cyber recovery as an operational control, not a backup feature. In critical infrastructure, the failure mode is not just data loss. It is restoring compromised identity, malware, or configuration state back into systems that have to keep essential services running.

The key governance question is whether recovery can reintroduce trust safely after an incident. That matters to NHI governance because domain controllers, service accounts, and application servers often carry the same identity debt that existed before the incident, unless restoration is isolated and validated before use.

Key questions

Q: What breaks when recovery systems are treated as passive backups instead of trusted environments?

A: The main failure is that contaminated identity, malware, or configuration state can be restored along with the data. That turns recovery into re-infection. Security teams should validate restored systems in an isolated environment, confirm privilege integrity, and only then allow cutover into production.

Q: Why do air-gapped backups still require privileged access controls?

A: Because the backup plane still has administrators, orchestration accounts, and restore permissions that can be abused. If those identities are over-privileged or shared, attackers can tamper with recovery itself. The control objective is to separate duties and tightly govern who can initiate or approve restores.

Q: How can organisations tell whether recovery testing is actually working?

A: Look for evidence that restore tests prove clean execution, not just successful startup. The test should confirm malware scanning, access integrity, dependency health, and whether critical systems can return without reintroducing compromised identities or unsafe configuration.

Q: Who should approve a restore when critical systems are involved?

A: Approval should sit with a defined recovery authority, not with whoever happens to hold production admin rights. For critical systems, organisations need clear separation between operational restoration, security validation, and business cutover so no single identity controls the entire recovery path.

Technical breakdown

Why cleanroom recovery changes the trust boundary

Cleanroom recovery places restored systems in an isolated environment so they can be validated before rejoining production. That matters because a backup is not automatically clean simply because it is offline or immutable. Malware, embedded credentials, and misconfigurations can all be preserved inside restore points. Cleanroom designs therefore create a separate verification layer for integrity, malware scanning, and operational readiness before business systems are trusted again.

Practical implication: treat recovery infrastructure as a controlled trust boundary and require validation before any restored identity or server is allowed back into production.

Why air-gapped backups still need identity governance

Air-gapped backups reduce exposure, but they do not remove identity risk. Administrative access to backup systems, recovery orchestration accounts, and privileged restore workflows can still become an attack path if credentials are shared, over-privileged, or poorly monitored. In practice, the recovery plane becomes another identity surface that needs least privilege, separation of duties, and clear ownership.

Practical implication: inventory the accounts, tokens, and roles that can reach backup and recovery systems, then apply privileged access controls to them.

Why restore testing must prove more than availability

Testing a recovery platform should verify more than whether systems start. It should prove that restored domain controllers, SQL servers, and application servers can come back without reintroducing malicious state or broken dependencies. Annual validation is useful, but the real objective is to confirm that the team can restore a known-good environment fast enough for operational needs while preserving security assumptions.

Practical implication: test recovery with identity, malware, and dependency checks, not just with a successful boot or failover event.

NHI Mgmt Group analysis

Clean recovery is now an identity governance problem as much as a backup problem. SMMPA’s requirements show that recovery cannot be evaluated only by uptime or data durability. If restored systems bring back compromised access paths, the organisation has only recreated the incident in a new environment. Practitioners should treat restore workflows as part of the identity control plane, not a separate infrastructure concern.

Air-gapped backups reduce exposure, but they do not eliminate privileged access risk. A backup environment still depends on administrative accounts, orchestration permissions, and restore approvals. Those identities are high-value because they can bypass normal production controls at the exact moment the business is most vulnerable. The lesson for NHI governance is that recovery accounts deserve the same lifecycle discipline as production credentials.

Recovery trust boundary: the decisive control is no longer where the backup lives, but whether restored state is proven clean before it is trusted. That concept captures the shift this case represents. Cleanroom-style recovery is valuable because it separates preservation from re-entry, which is the distinction many programmes miss when they equate backup success with recovery safety. Practitioners should redesign recovery to validate trust before restore completion.

Insurance requirements are increasingly shaping control design, but compliance language should not obscure operational reality. Air-gapped backups and malware scanning at rest are only meaningful if the underlying restore process prevents contaminated identity state from returning to production. The governance question is whether recovery testing includes the systems that actually authenticate and authorize work. Practitioners should review recovery plans through the lens of identity continuity, not just storage resilience.

From our research:

• 91% of former employee tokens remain active after offboarding, leaving organisations vulnerable to potential security breaches, according to The 2025 State of NHIs and Secrets in Cybersecurity.
• 62% of all secrets are duplicated and stored in multiple locations, causing unnecessary redundancy and increasing the risk of accidental exposure, according to The 2025 State of NHIs and Secrets in Cybersecurity.
• For a broader breach-pattern lens, see The 52 NHI breaches Report for recurring identity failure modes across real incidents.

What this signals

Recovery governance is becoming an identity programme concern, not just an infrastructure one. As restore environments become more controlled, teams need to know which identities can authorize cutover, who can validate the trust boundary, and where privileged access is still standing. The practical signal is that recovery exercises should be reviewed with the same scrutiny as privileged access reviews, because recovery authority is now a security control.

The next maturity step is to make restore trust measurable. That means proving that cleanroom validation checks for compromised credentials, malicious persistence, and unsafe dependencies before production re-entry. Teams that cannot demonstrate that control are still assuming backups are safe by default, and that assumption is increasingly weak.

Recovery trust boundary: organisations should treat it as a named control surface and track it separately from backup durability. For deeper breach-pattern context, the recurring failure is often not the backup itself but the identity state that travels with it, which is why lifecycle and privileged-access governance remain relevant long after the incident ends.

For practitioners

• Map recovery-plane identities Identify every account, token, role, and service principal that can operate backup, replication, or restore workflows. Remove shared admin access and document who can authorize a cleanroom restore.
• Validate restore cleanliness before cutover Require malware scanning, credential integrity checks, and dependency validation before any restored domain controller, database, or application server is allowed back into production.
• Test recovery with identity failure scenarios Run exercises that assume compromised credentials, not just damaged infrastructure. Include restore-path access review, approval bypass checks, and the behaviour of privileged accounts during failover.
• Separate backup administration from production privilege Use distinct administrative roles for backup management and live environment operations, and review them on a fixed cadence so restore authority does not become standing privilege.

Key takeaways

• Cleanroom recovery changes the problem from storing backups to proving that restored systems are safe to trust.
• The evidence in SMMPA’s use case is that critical infrastructure recovery now depends on air-gapped design, malware scanning, and repeated validation.
• Practitioners should govern the backup plane as an identity surface, because restore authority and recovery trust are now part of the security boundary.

Key terms

• Cleanroom Recovery: A recovery model that restores systems into an isolated environment before they are trusted in production. It is designed to verify integrity, check for malware, and confirm dependencies before business systems are allowed back online.
• Recovery Trust Boundary: The line between preserving data and reintroducing it into operations. In identity-heavy environments, that boundary must account for credentials, permissions, and system state, because a restore can bring back compromise as easily as it brings back availability.
• Air-gapped Backup: A backup copy that is separated from production systems to reduce direct attacker reach. The control reduces exposure, but it does not eliminate the need to govern the identities and permissions that can access, restore, or modify the backup environment.

What's in the full article

Commvault's full case study covers the operational detail this post intentionally leaves for the source:

• How SMMPA configured Cleanroom Recovery for its VMware environment in Azure.
• The specific systems SMMPA prioritised for restoration, including domain controllers, SQL servers, and application servers.
• How the team aligned the recovery design with cyber insurance requirements for air-gapped backups and malware scanning at rest.
• What the annual testing process looks like in practice and how the team plans to increase test frequency.

👉 The full Commvault case study covers the deployment details, validation approach, and recovery priorities.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM or identity security programme, it is worth exploring.