Infrastructure as code platform rollout playbook: workspaces, runners, policy gates and pilot waves

Most infrastructure as code platforms are rolled out team by team with no shared design, so the organisation ends up with hundreds of workspaces, runners that can reach every account and policy gates nobody dares to turn on. This playbook covers the rollout of a central IaC platform as a programme: how to structure workspaces and projects so blast radius follows environment and ownership, where to place runners and how to isolate them, how to stage policy gates from advisory to blocking, and how to move teams across in pilot waves without stopping delivery. It is vendor-neutral and applies to self-hosted and SaaS orchestration alike. It is written for platform engineering leads, cloud security architects and the change owners who sign off production access.

This page does not repeat the credential and policy detail held elsewhere. State backends and provider credentials are covered in the Terraform and IaC Pipeline Credential Security Guide, the identity rules to enforce in the gate are in the IaC identity policy checks, and what to do if the platform is breached is in If your infrastructure as code platform is compromised.

Key takeaways

• Design the workspace map before onboarding anyone. One workspace per component per environment, owned by a named team, with production in a separate project, is far easier to secure than a migration of whatever structure each team already has.
• Runner placement is the real security boundary. A runner that applies to production should serve only production workspaces, sit in a network zone that reaches only production endpoints, and be reset after every run.
• Split plan from apply. Plans run on every pull request from untrusted branches; applies run only after review, on separate runners with separate identities.
• Stage policy gates in three steps. Advisory first to measure, then blocking on new code for a short critical list, then blocking everywhere with a governed exception path.
• Migrate in waves and close the side doors. The platform only reduces risk once console and laptop applies to the same accounts are removed, so every wave ends with direct write access revoked.

What a central IaC platform has to provide

An IaC platform is the orchestration layer around the IaC tool: it stores state, runs plan and apply jobs, holds or brokers the credentials that reach cloud and SaaS APIs, applies policy to plans, and records who approved what. NIST SP 800-204D treats infrastructure code as one of the artefact types whose pipeline needs defined security requirements, and its GitOps requirements apply directly: automate rather than change by hand, preserve the data for every release, make changes only through code, and monitor for drift between the repository and the running state.

Before choosing structures, agree the non-negotiable outcomes the rollout must deliver. These become the exit criteria of each wave.

The two-person rule comes straight from the NSA and CISA guidance on defending CI/CD environments, which says no single developer should be able to check in code without another developer reviewing and approving it, and asks for separation of duties between source code access and changes to the build environment.

Designing workspace and project structure

A workspace (called a stack, project or environment in different tools) is the unit that owns a state file, a set of variables, a credential and a run history. Its boundaries decide how much one compromised run can change, so draw them deliberately.

• Split by environment first: production, pre-production and development never share a workspace, a state file or a credential. Use a separate project or organisation for production so team membership there is reviewed on its own.
• Then by component and owner: one workspace per deployable component (network core, identity, shared data, each application). A workspace with more than one owning team will drift into shared admin rights.
• Keep foundations apart: the landing zone, organisation policies, identity federation and logging live in a small set of tier 0 workspaces with their own approvers. See the Cloud Landing Zone Security Guide for what belongs there.
• Share outputs, not state: where one workspace needs values from another, publish explicit outputs with read-only access rather than granting read on the whole remote state, which contains every attribute of every resource.
• Name for automation: a pattern such as <org>-<component>-<env> lets policy, runner selection and access rules key off the name or tags instead of hand-maintained lists.

Create workspaces through code as well. A workspace vending module that sets the runner pool, approval rules, policy sets, variable sets and the bound identity removes the manual settings that drift first.

Runner placement and isolation

The runner is where provider credentials are used and where plan output is rendered, so it is the place an attacker wants to execute code. OWASP lists this as CICD-SEC-5 Insufficient PBAC (pipeline-based access controls) and recommends sharing execution nodes only between pipelines of the same sensitivity, scoping secrets to the pipeline and step that need them, returning nodes to a clean state after every run, and restricting their network access. NIST SP 800-204D asks for an isolated CI/CD environment kept separate from elevated administrator credentials used for cloud deployments. General runner hardening is covered in Self-hosted CI/CD Runners and Build Agents; the IaC-specific placement decisions are below.

• Place apply runners inside the target zone: a production runner in the production network reaches private endpoints without opening inbound paths, and cannot reach development.
• Never let pull request code run on apply runners: plan executes provider and module code, including external data sources, so an unreviewed branch can run commands. Route all pre-merge work to the plan pool.
• Pin the toolchain: runner images carry the IaC binary, providers and modules from an internal mirror, with checksums locked in the repository, so a run cannot fetch an unreviewed provider at apply time.
• Treat SaaS-hosted runners as a separate trust decision: where the vendor runs the job, the vendor's isolation model and network origin replace yours. Many organisations keep vendor-hosted runners for plans and use their own runners for production applies.

SLSA Build Level 3 sets a useful target for the production pool: the platform should stop runs from influencing one another even within the same project, and keep signing material out of reach of user-defined steps.

Where identity fits

Every workspace is a non-human identity with write access to infrastructure, and every runner pool is a place those identities are used. The rollout should leave a clean map: one plan identity and one apply identity per workspace, each issued per run through workload identity federation, bound to the workspace and branch claims, and owned by the team that owns the workspace. The platform's own administrators, the identity that registers runners and any API token used to create workspaces are tier 0 credentials. Count them, vault them and review them quarterly. Detail on trust conditions and static key removal is in the Terraform and IaC Pipeline Credential Security Guide and the Workload Identity Federation Guide.

Staging policy gates

NIST SP 800-204D describes build policies enforced by a policy engine and deployment verification policies that allow or block a release. In an IaC platform that is the policy set evaluated against every plan. Turning it on at full strength on day one blocks unrelated changes on legacy findings, and teams then route around the platform. Stage it.

• Start the critical list small: public storage, wildcard trust policies, new long-lived access keys, disabled encryption on state and log stores, and changes to logging or organisation guard rails. The identity rules and how to write them are in the IaC identity policy checks.
• Version the policy set like code: policies live in their own repository with code owners from security, tests against known good and known bad plans, and a release process. A broken policy release is rolled back by pinning the previous version.
• Make overrides visible: every override records the run, rule, approver and reason, and feeds a weekly report. An override that is used repeatedly for the same rule signals a bad rule or a missing pattern.
• Put the gate where it cannot be skipped: evaluate on the platform after plan and before apply, not only in a pull request check that a team can remove from its own pipeline.

Pilot waves: the rollout sequence

Move teams across in waves. Each wave reuses the templates proven in the previous one and ends with the old path closed.

1. Wave 0, foundations (platform team only): deploy the platform, the state store and the runner pools; build the workspace vending module; connect run logs to the SIEM; bring the platform's own configuration under code. Prove that a test workspace can plan and apply to a sandbox account with a federated identity and no stored keys.
2. Wave 1, pilot (one or two willing teams, non-production): onboard one simple and one complex component. Import existing resources into state rather than recreating them, and run every policy in advisory mode. Measure plan time, failure causes and how often engineers still need console access.
3. Wave 2, pilot production: move the pilot teams' production workspaces to the production apply pool with approvals enforced. Turn on blocking for the critical list for these workspaces. Remove the teams' direct production write roles and leave a documented break-glass path; see the Break-Glass Account Guide.
4. Wave 3 onwards, scale by risk: onboard teams in order of production exposure, not enthusiasm. Each wave has a fixed exit checklist: workspaces created from the module, identities federated, static keys disabled, direct write access removed, drift report clean for two weeks.
5. Final wave, tier 0: move the landing zone and identity workspaces last, once runner isolation and approval rules have been proven, onto the tier 0 pool with its own approvers.
6. Close the estate: deny write actions to platform-managed accounts for any principal other than the apply identities and break-glass roles, using organisation-level guard rails where the cloud supports them, and alert on any exception.

Testing: before each wave, run a plan against a copy of the workspace with the new runner pool and policy set and compare the output with the old path; a plan that shows changes the team did not expect means state or module versions differ. Rollback: keep the old pipeline disabled but intact for one release cycle per team; if a wave fails, re-enable it for that team only, with its static credential re-enabled for a fixed period, and fix the cause before re-migrating. Never re-grant broad console write access as the rollback.

Decision guidance

• Self-hosted or SaaS orchestration: SaaS removes platform patching but moves state storage, run execution and audit logs into a third party; check data residency and whether you can bring your own runners for production. Self-hosted gives network control but makes the platform a tier 0 system you must patch, back up and monitor.
• Small organisations: one project per environment, a shared non-production runner pool, a dedicated production runner and five or six blocking rules give most of the value. Skip the separate tier 0 pool and instead require two approvers on foundation workspaces.
• Large or regulated organisations: per-business-unit projects, per-environment runner pools in each network zone, a policy release process with security code owners, and evidence exported per run for change audits. Map the approval record to your change process; see Change Management Security.
• Multi-cloud: keep one workspace model and one policy repository, with cloud-specific rule packs, rather than one platform per cloud.

Detection and measurement

Log at the platform, runner and cloud layers. Platform logs should record workspace and policy changes, team and token administration, run approvals and overrides. Runner logs show which job ran on which instance and what it reached. Cloud audit logs show which identity made each change, which is how you find writes that did not come from an apply identity.

• Coverage: share of in-scope cloud accounts whose write activity comes only from platform apply identities and break-glass roles. Target 100% for production.
• Drift: number of workspaces with unreconciled drift older than seven days.
• Policy override rate: overrides per 100 production applies, tracked per rule.
• Runner hygiene: share of runs on fresh instances, and number of runner pools serving more than one environment (target zero).
• Static credentials: count of workspaces still using stored long-lived keys, falling to zero by the end of the rollout.
• Wave velocity: workspaces onboarded and direct write roles removed per month, against plan.

High-value alerts: a production apply without a recorded approval, a change to a policy set or runner pool assignment outside the platform repository, a new platform admin or API token, and a write to a managed account by any identity that is not an apply identity.

Framework mapping

Common failure patterns

• Lifting and shifting team pipelines: each team's old structure, shared credentials and exemptions are imported unchanged, and the platform inherits all of them. Use the vending module, even for migrated workspaces.
• One runner pool for everything: pull request plans from any branch share instances and network reach with production applies, so a malicious module in a branch can reach production credentials.
• Gates that only advise, forever: stage 1 never ends because no one owns the exit criteria. Put stage dates in the programme plan.
• Leaving the side door open: teams keep console write roles "for emergencies", and the drift those changes cause makes the next apply destructive. Replace them with a logged break-glass path.
• Full remote state as the integration API: workspaces read each other's entire state to get one value, so a low-tier workspace credential exposes production attributes.
• Platform admins outside identity governance: platform owner roles and API tokens are created during wave 0 and never reviewed, leaving the most powerful accounts in the estate unaudited.

Practitioner checklist

• Agree the five rollout outcomes and turn them into an exit checklist for every wave.
• Publish the workspace naming and structure standard, with production in its own project.
• Create every workspace from a governed module that sets runner pool, approvals, policies and identity.
• Run separate plan, non-production apply, production apply and tier 0 runner pools, each reset after every job.
• Route all pre-merge plans to the plan pool with read-only identities.
• Stage policy gates from advisory to blocking with dated exit criteria and a security-owned policy repository.
• Migrate in waves ordered by production risk, removing direct write access at the end of each wave.
• Forward platform, runner and cloud audit logs to the SIEM and alert on writes from non-apply identities.
• Review platform administrators, runner registration credentials and API tokens every quarter.

Standards and references

• NIST: SP 800-204D, Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
• NSA and CISA: Defending Continuous Integration/Continuous Delivery (CI/CD) Environments
• OWASP: Top 10 CI/CD Security Risks
• OWASP: CICD-SEC-5 Insufficient PBAC (Pipeline-Based Access Controls)
• OWASP: CICD-SEC-1 Insufficient Flow Control Mechanisms
• SLSA: Security levels, version 1.0
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• NIST: Cybersecurity Framework 2.0

Related NHI Mgmt Group resources: Terraform and IaC Pipeline Credential Security Guide, IaC identity policy checks, If your infrastructure as code platform is compromised, Self-hosted CI/CD Runners and Build Agents, CI/CD Pipeline Identity Security Guide.