Configuration Management Tool Rollout Playbook

A configuration management tool is built to run privileged changes on every server it manages, so a rollout that treats it as a convenience script rather than a tier zero system hands that power to whoever reaches its controller, its code repository or its credentials. MITRE ATT&CK lists configuration management and deployment suites under T1072 Software Deployment Tools because access to them can mean remote code execution across every connected system. This vendor-neutral playbook is for infrastructure, platform and security engineers introducing a configuration management tool or replacing ad hoc scripts with one. It covers the design decisions, hardening the control plane, the code and review pipeline, the identities the tool holds, a pilot in dry-run mode, rollout waves with rollback, and the metrics that show it is working. Day-two hardening is covered in Securing the configuration management tool itself, and incident response in If your configuration management tool is compromised.

Key takeaways

Classify the tool as tier zero from day one. Its controller, code repository, secrets store and agent trust root together can change every managed host, so they need the same protection as domain controllers or the identity provider.
The repository is the real control plane. Whoever can merge to the branch the tool applies can run code as root or SYSTEM everywhere, so branch protection, mandatory review and signed commits matter more than the console login.
Design the credentials before the first playbook. Decide how the tool authenticates to hosts (SSH keys or certificates, WinRM, agent certificates) and how it reads secrets, so no password or key lands in the repository.
Pilot in report-only mode. Run in dry-run or audit mode against a pilot ring, compare intended and actual changes, then enforce in rings with a tested rollback.
Measure drift and blast radius, not just coverage. Track hosts under management, drift corrected, failed runs, unreviewed changes and how many hosts any single identity can change.

Phase 0: design decisions that shape security

Configuration management tools fall into two broad architectures, and the choice drives where credentials live and what has to be protected. Some products support both.

Decision	Push model	Pull (agent) model	Security consequence
Who starts the connection	Controller connects out to hosts over SSH or WinRM	Agent on each host checks in to a server or reads from a repository	Push needs inbound management ports on every host; pull needs inbound access only to the server
Credential on the host side	A privileged account and its key or password on every host	An agent certificate or token issued at enrolment	Push concentrates reusable credentials on the controller; pull concentrates a certificate authority on the server
Trust root to protect	Controller's SSH keys or SSH certificate authority, WinRM credentials	The server's certificate authority and enrolment (autosign) policy	A permissive enrolment policy lets any machine join and receive configuration, including secrets
Offline hosts	Missed until the next push	Converge when they next check in	Pull gives steadier drift correction
Agentless or agent	No resident agent to patch	Agent runs as root or SYSTEM and must be patched	Agent vulnerabilities become fleet-wide; keep agent versions in the inventory

Record four other decisions before building anything:

Scope boundaries: one controller per environment or security zone (production, non-production, payment card or OT segments) limits blast radius. A single controller for everything means one compromise reaches everything.
Source of truth: which repository and branch the tool applies, and whether changes flow by merge (GitOps style) or by an operator triggering a run. See the GitOps Security Guide.
Secrets integration: whether the tool reads from an external secrets manager at run time or uses its own encrypted store. External lookup keeps secrets out of the repository and gives an audit trail.
Baselines to enforce: CIS Benchmarks or an internal standard derived from them, mapped to NIST SP 800-53 CM-6 Configuration Settings. NIST SP 800-128 frames this as security-focused configuration management. See the NIST SP 800-128 Security-Focused Configuration Management Guide.

Phase 1: build and harden the control plane

The ATT&CK mitigations for T1072 are a good build checklist: limit access to deployment systems to a small number of administrators (M1026 Privileged Account Management), isolate them with firewalls (M1030 Network Segmentation), require MFA (M1032), use unique credentials not reused elsewhere (M1027), patch the deployment system itself (M1051), and keep signing certificates off the deployment system where only signed code may be deployed (M1029).

Component	Hardening before first production run
Controller or server host	Dedicated, minimal host built from your own baseline; management network only; no general user logins; disk encryption; patched on a defined cycle
Web console and API	Single sign-on with phishing-resistant MFA, no local accounts except a vaulted break-glass account, role-based access by environment, API tokens short-lived and scoped
Agent enrolment	Manual or policy-based certificate approval tied to the asset inventory; no blanket autosign; certificate revocation tested
Network paths	Only the controller may reach SSH or WinRM on managed hosts; agents reach only the server port; deny everything else at host and network firewalls
Logging	Console logins, API calls, job launches, who triggered each run, and full run output shipped to the SIEM, with retention aligned to your audit policy
Backups	Controller configuration, certificate authority keys and encrypted stores backed up offline, with a restore tested before go-live

Administrators should reach the console and controller from privileged access workstations, not daily-use laptops. See the Privileged Access Workstations Guide.

Phase 2: the code and review pipeline

Once the tool is live, a merged pull request is a privileged change to production. Treat the configuration repository like production code and apply NIST SP 800-53 CM-3 Configuration Change Control with CM-3(2), testing and validating changes before they are finalised.

Protect the branch the tool applies: require pull requests, at least one reviewer from outside the author's team for production, passing checks, and no force pushes. Restrict who can change these protections.
Sign commits and verify them: require signed commits on protected branches so a stolen repository token cannot silently push. NIST SP 800-53 CM-14 Signed Components and SI-7(6) cover the cryptographic integrity idea.
Scan every change: run linting, secret scanning and infrastructure-as-code policy checks in the pipeline, and block merges that add plaintext credentials, disable host firewalls or open management ports to the internet. See the Checkov and IaC Scanners Guide.
Pin third-party content: community roles, modules and collections run with the tool's privileges. Mirror them internally, pin versions and review updates like any other dependency.
Keep previous configurations: CM-2(3) Retention of Previous Configurations is what makes rollback possible; tag every applied release.

The Change Management Security guide covers approval design in more depth.

Where identity fits: the credentials the tool holds

A configuration management tool is one of the most privileged non-human identities in an estate, and it usually holds several others. Plan each one deliberately.

Identity	Common mistake	Better pattern
Controller to host (SSH)	One long-lived private key in every host's authorized_keys, copied to engineers' laptops	SSH certificates with short validity issued to the controller, or a key held only on the controller and rotated; see the SSH Protocol and Certificate Authority Guide
Controller to Windows hosts	A domain admin account used for WinRM	A dedicated account per environment with only the rights needed, never a domain admin, managed in a PAM vault
Privilege escalation on hosts	Unrestricted passwordless sudo for the management account	Sudo scoped to what the tool runs, logged, reviewed per environment
Agent certificates	Any host can enrol and receive configuration	Enrolment tied to inventory, certificates revoked when hosts are retired
Secrets in configuration	Passwords and API keys committed to the repository, even encrypted with a shared key in the same repo	Run-time lookup from a secrets manager using the controller's own workload identity
Cloud credentials	Static access keys for provisioning modules	Federated, short-lived credentials scoped per environment
Repository and pipeline tokens	A personal access token belonging to one engineer	A dedicated machine identity with read-only access to the repository it applies

The Internet Archive breach shows why repository hygiene matters: an access token exposed in a configuration file opened the source code and further credentials. The planned Configuration Management Tool Identity guide goes deeper on this layer, and the Secrets manager rollout playbook covers the secrets side.

Phase 3: pilot in report-only mode

Most configuration management tools offer a dry-run, check or no-op mode that reports what would change without changing it. Use it before any enforcement.

Choose a pilot ring: 20 to 50 non-production hosts covering each operating system and role, plus a few low-risk production hosts owned by the platform team.
Run in report-only mode for at least one change cycle: review every reported change with host owners. Unexpected changes usually mean the baseline is wrong for that role, not that the host is wrong.
Enforce on the pilot ring: apply, then verify with an independent scanner (for example a CIS Benchmark assessment) that the intended settings are present.
Test failure behaviour: break a run deliberately (syntax error, unreachable secrets store) and confirm the tool fails closed and alerts, rather than applying half a configuration.
Rehearse rollback: revert to the previous tagged release and confirm the hosts converge back. Some changes, such as removed packages or rotated keys, cannot be reverted by re-applying, so document compensating steps for them.

Phase 4: rollout waves, exceptions and rollback

Expand in rings, for example pilot, all non-production, low-risk production, then critical production, with a hold period at each step and a go or no-go review using the metrics below. Bring hosts under management in report-only mode first, then switch to enforcement per role once drift reports are clean.

Exceptions: record every deviation from the baseline with an owner, reason and expiry, in the repository next to the code, so the exception is versioned and reviewed. CIS Control 4 Safeguard 4.1 expects a documented secure configuration process reviewed at least annually.
Change windows: for critical systems, enforce only within agreed windows or require a manual approval step before production runs.
Rollback: the default rollback is re-applying the previous tag. Keep the ability to pause enforcement per host group, and make sure pausing is itself logged and alerted.
Retire old scripts: remove the cron jobs, scheduled tasks and shared keys the tool replaces; otherwise two systems fight over the same settings and the old credentials remain.

The Configuration Drift and Baseline Enforcement Guide covers baseline choice and exception handling in more detail.

Small versus large organisations

Aspect	Small estate (tens of servers)	Large estate (thousands)
Architecture	Agentless push from one hardened controller is usually enough	Controllers per zone, or pull agents with regional servers
Review	Two-person rule on merges, even within one team	Code owners per role, separate approvers for production
Credentials	Vaulted keys rotated on a schedule	SSH certificates, federated cloud credentials, run-time secrets lookup
Rollout	Pilot plus one or two waves over weeks	Ring-based by business unit over months
Main risk	One administrator's laptop holds the master key	Too many people and pipelines can trigger production runs

Detection and measurement

Send console and API audit logs, job history and run output to the SIEM. Alert on runs triggered outside the pipeline, ad hoc command execution against many hosts, new administrator accounts or API tokens, changes to branch protection, agent certificate approvals for hosts not in the inventory, and SSH or WinRM sessions to managed hosts from anything other than the controller. ATT&CK T1098.004 SSH Authorized Keys is a useful detection reference: an unexpected key appearing in authorized_keys files across many hosts is a strong signal.

Coverage: percentage of in-scope hosts under management, compared with the asset inventory (CM-8).
Baseline compliance: percentage of managed hosts passing the enforced baseline, confirmed by an independent scanner.
Drift corrected: number of settings reverted per week, trending down as teams stop making manual changes.
Unreviewed changes: production runs without a linked approved pull request (target zero).
Credential exposure: secrets found in the repository by scanning (target zero) and age of the controller's host credentials.
Blast radius: the largest number of production hosts any single human or machine identity can change without a second approver.

Framework mapping

Rollout activity	NIST SP 800-53 Rev. 5	NIST CSF 2.0	Other references
Baseline definition and automated enforcement	CM-2, CM-2(2), CM-6, CM-6(1)	PR.PS-01	CIS Controls v8.1 Control 4; NIST SP 800-128
Change control and testing	CM-3, CM-3(1), CM-3(2), CM-2(3)	PR.PS-01	Change management policy
Who can change the tool and repository	CM-5, CM-5(1), AC-6	PR.AA-05	ATT&CK T1072 mitigations M1026, M1032
Signed code and integrity	CM-14, SI-7, SI-7(6)	PR.PS-01	ATT&CK M1029
Tool credentials and secrets	IA-5, AC-6	PR.AA-01	ATT&CK T1552.001 Credentials In Files
Inventory and coverage	CM-8	ID.AM-02, ID.AM-08	CIS Controls v8.1 Controls 1 and 2
Logging and monitoring of the tool	AU-2, AU-12	PR.PS-04, DE.CM-09	ATT&CK T1098.004 SSH Authorized Keys

Common failure patterns

Autosign left on: enrolment accepts any machine that asks, so an attacker's host joins and receives configuration, including any secrets embedded in it.
Secrets in the repository: passwords and keys committed in plaintext, or encrypted with a key stored alongside them, then exposed when the repository or a token leaks.
Domain admin as the run account: the Windows management account is a domain administrator, so a controller compromise is an Active Directory compromise.
Unprotected main branch: engineers push directly to the branch the tool applies, so one stolen token is fleet-wide code execution.
Controller exposed to the internet: master or API ports reachable from outside. Several configuration management server vulnerabilities are listed in the CISA Known Exploited Vulnerabilities catalog, including authentication bypass and path traversal flaws in a server process (CVE-2020-11651 and CVE-2020-11652).
Old scripts never retired: legacy cron jobs and shared keys keep running beside the new tool, so drift and credential sprawl continue.

Practitioner checklist

Classify the controller, repository, secrets store and agent certificate authority as tier zero assets.
Choose push or pull deliberately and document the trust root you must protect.
Separate controllers or scopes by environment and security zone.
Put the console behind single sign-on with phishing-resistant MFA and vault the break-glass account.
Disable blanket agent autosign and tie enrolment to the asset inventory.
Protect the applied branch, require review and signed commits, and scan every change for secrets.
Use run-time secrets lookup and short-lived or certificate-based host credentials.
Pilot in report-only mode, verify with an independent scanner and rehearse rollback.
Roll out in rings with go or no-go reviews and versioned, expiring exceptions.
Alert on runs outside the pipeline, new admin tokens, enrolment of unknown hosts and unexpected SSH keys.

Configuration management tool rollout playbook: phased implementation with security built in