A compromised configuration management tool is a ready-made way to run code as root or SYSTEM on every server it manages, using credentials and change channels your teams already trust. These tools hold SSH keys, privileged service accounts, encrypted secret stores, cloud credentials and an agent certificate authority, and they apply whatever the code repository tells them to. This guide is a response playbook for a suspected or confirmed compromise of the controller, its agents, its code repository or its secrets. It covers how the incident shows up, triage, containment in priority order, credential rotation, re-keying the agent trust, rebuild and evidence. It is written for incident responders, infrastructure and platform engineers, and it is vendor-neutral, covering push-based tools that connect out over SSH or WinRM and pull-based tools whose agents check in to a server.
Key takeaways
- Stop the next run before anything else. Pausing scheduled runs, agent check-ins and pipeline-triggered applies stops the attacker using the tool as a deployment system (ATT&CK T1072 Software Deployment Tools).
- The repository is part of the blast radius. A malicious change to a role, module, state file or template is reapplied on every run until it is removed from source control.
- Re-key the trust, not only the passwords. If the agent certificate authority or the controller's SSH keys were exposed, every node's trust in the controller has to be re-established.
- Every secret the tool could decrypt is exposed. Encrypted variables, secret store tokens and cloud keys used during runs must be rotated at their owners.
- Managed nodes need a hunt, not an assumption. Compare node state with the reviewed desired state and rebuild nodes with unexplained changes.
Why this compromise is different
A configuration management tool is designed to make privileged changes at scale with little human review per node. That design is the risk. The two common architectures fail differently, which changes containment.
| Aspect | Push model (controller connects to nodes) | Pull model (agents check in to a server) |
|---|---|---|
| Transport | SSH (TCP 22) for Linux, WinRM (TCP 5985 HTTP, 5986 HTTPS) for Windows | Agent to server over TLS with client certificates issued by the server's own CA |
| Credentials held centrally | Private SSH keys, privileged Windows accounts, become or sudo passwords | CA private key, server certificate, node certificates, encrypted data keys |
| How the attacker executes | Runs an ad hoc job or modified playbook against an inventory | Changes the code or data the server compiles for nodes; agents apply it on next check-in |
| Fast containment lever | Revoke controller keys on nodes, block controller egress to node ports | Stop the server or block agent port, disable agent runs on nodes |
| Trust to re-establish | Authorised keys and accounts on every node | Node certificates and, if exposed, the whole CA |
Typical entry routes are a stolen administrator session or API token on the controller's web interface, a compromised developer account with write access to the configuration repository, an exploited server vulnerability, or a poisoned third-party module (ATT&CK T1195.002). The CISA Known Exploited Vulnerabilities Catalog lists, for example, CVE-2020-11651 (authentication bypass) and CVE-2020-11652 (path traversal) in a widely used configuration management master, both of which allowed remote attackers to reach the component that commands every managed node. See the Securing the configuration management tool itself guide for the preventive baseline.
How the compromise shows up
| Signal | Where to look | What it may mean |
|---|---|---|
| Ad hoc jobs or runs outside change windows, or against unusual inventories | Controller job history and audit log | Interactive use of the tool by an attacker (T1072) |
| Commits to roles, modules or states without a reviewed merge request | Repository audit log, branch protection bypass events | Planted persistence in desired state |
| New users, API tokens or credentials added to the controller | Controller audit log, identity provider admin logs | Persistence (T1098 Account Manipulation) |
| Unexpected new local accounts, authorised keys, services or scheduled tasks on many nodes at once | Host logs, EDR, file integrity monitoring | Mass deployment of persistence (T1543 Create or Modify System Process) |
| Node certificate requests signed outside normal provisioning | CA signing log on the server | Rogue node or attacker-held certificate |
| Secret store reads by the tool's identity outside run times | Secret store audit log | Credential harvesting with the tool's token (T1552) |
Triage questions
- Which component is affected? Controller or server, its web interface or API, the configuration repository, third-party modules, the secret store integration or individual nodes.
- How many nodes could it reach, and with what rights? Export the inventory and the credentials or certificates used for each group before changing anything.
- What ran, where and when? Pull job history and agent run reports for the suspected window and list every node that applied a change.
- Is the desired state clean? Diff the repository against the last reviewed release and check commit signatures and merge records.
- What could the tool decrypt or fetch? List encrypted variables, data keys, secret store paths and cloud credentials used during runs.
- Do the tool's own logs live elsewhere? If job history exists only on the controller, treat it as possibly altered and look for a copy in the SIEM.
Containment by priority
- Preserve: image the controller or server (memory and disk where you can), export job history, repository history, audit logs and the CA signing log.
- Stop runs: disable schedules and pipeline triggers, put the controller into maintenance or stop the service, and on pull-model nodes disable the agent run service or block the agent port at the network.
- Freeze the repository: restrict write access to the response group, enforce branch protection and suspend deploy keys used by automation.
- Cut controller reach: block the controller from SSH and WinRM to nodes except for a named response path, and remove its keys from authorised keys files where you can do so through another channel.
- Restrict the tool's secret access: revoke the tool's secret store token and cloud credentials so it cannot fetch more secrets while you investigate.
- Use an alternative change path: agree how urgent changes will be made while the tool is down, for example a small, separately credentialed admin path with full logging.
Decision: stop the tool completely or run it in a restricted mode. Stop it completely when the controller, CA or repository is confirmed compromised. A restricted mode (read-only reporting, no applies) is reasonable only when the compromise is limited to one user's web session and the audit trail is independent and intact.
Small and large organisations: with a few hundred nodes and one controller, a full re-key and node-by-node verification is achievable in days and is the safer default. With tens of thousands of nodes across several controllers or server pools, scope first: identify which controllers, environments and inventories the attacker reached, contain those completely, and keep unaffected pools in check-only mode until their audit trails are verified. Large estates should also pre-agree how many node rebuilds per day operations can absorb, because that number sets the recovery timeline.
Credential rotation and re-keying
| Order | What to rotate | How | Verify |
|---|---|---|---|
| 1 | Controller admins, API tokens, repository deploy keys and CI tokens | Re-issue from SSO, scope tokens per job, delete unused tokens | Token inventory matches the approved list |
| 2 | Agent CA and node certificates (pull model) | Stand up a new CA on the rebuilt server, re-enrol each node through a verified channel, revoke and distrust the old CA | Nodes present only certificates from the new CA |
| 3 | Controller SSH keys and Windows run accounts (push model) | Generate new keys on the rebuilt controller, replace authorised keys on nodes, prefer short-lived SSH certificates | Old keys absent from every authorised keys file |
| 4 | Encrypted variable keys and data keys | Re-encrypt with new keys and rotate the secrets they protected | Old keys destroyed; decryption with old keys fails |
| 5 | Secret store tokens and cloud credentials used by runs | Move to short-lived, workload-bound credentials; rotate any static key the tool held | Audit logs show only the new identity |
| 6 | Credentials the tool deployed to nodes (service account passwords, database users, API keys in templates) | Rotate at the owning system | Failed authentications with old values investigated |
Partial rotation is the most common way attackers regain access to trusted tooling, as the Trivy supply chain attack showed with an incomplete token rotation, and the Internet Archive breach showed with keys left unrotated after the first intrusion. See the Credential Rotation at Scale guide.
Eradication and rebuild
- Controller or server: rebuild from a clean image and verified installation packages. Do not restore the old server state; restore only reviewed code from the repository and re-create secrets.
- Repository: revert to the last reviewed release, review every change since then line by line, and pin third-party modules to verified versions or hashes.
- Nodes: for each node that applied a change during the window, compare its state with the reviewed desired state. Nodes with unexplained accounts, keys, services or binaries go to rebuild, starting with identity and crown-jewel systems. See the Server Build and Golden Image Hardening Guide.
- Staged return: re-enable the tool on a pilot group first, in check or dry-run mode, and confirm reported changes match expectations before enabling applies for wider rings.
- Rollback: keep the alternative change path ready, and be able to disable agent runs fleet-wide in minutes if pilot results show unexpected changes.
Evidence to preserve and notification
| Evidence | Why it matters |
|---|---|
| Controller image, job history and audit log | What the attacker ran and against which nodes |
| Repository history, merge records and signatures | How desired state was altered and by which identity |
| CA signing log and issued certificate list | Rogue nodes or attacker-held certificates |
| Secret store and cloud audit logs for the tool identity | Which secrets were read and must be treated as exposed |
| Node run reports and host logs | Scope of changes on managed systems |
Notification depends on what the managed systems hold. If the attacker could reach systems processing personal data or regulated services, assume reporting obligations may apply and involve legal and privacy teams early. Inform owners of systems whose credentials were deployed by the tool, because their rotation is part of your containment.
Where identity fits
The configuration management tool is itself one of the most privileged non-human identities in the estate, and it also creates and distributes other identities: service accounts, local admin passwords, database users and API keys in templates. An inventory of what the tool deploys, with owners, decides how fast rotation can finish. See the Configuration Management Tool Identity guide and the Service Account Security Guide.
Detection and measurement
- Time to stop runs: minutes from declaration to schedules, triggers and agent runs being disabled.
- Unreviewed change rate: commits reaching production branches without a reviewed merge, target zero.
- Re-key coverage: percentage of nodes trusting only the new CA or new controller keys.
- Secret rotation coverage: percentage of secrets the tool could decrypt or fetch that have been rotated at their owner.
- Drift findings: nodes whose state differs from reviewed desired state, each explained or rebuilt.
- Audit independence: percentage of controller job and audit events present in the SIEM.
Framework mapping
| Activity | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | MITRE ATT&CK |
|---|---|---|---|
| Triage and scoping | RS.MA-02, RS.AN-03 | IR-4, IR-5 | T1072, T1078 |
| Stop runs, freeze repository | RS.MI-01 | CM-3, CM-5, AC-6 | T1072, T1195.002 |
| Rotate and re-key | RS.MI-02 | IA-5, SC-12 | T1552, T1098 |
| Rebuild and compare to baseline | RC.RP-01, RC.RP-03, PR.PS-01 | CM-2, CM-6, SI-7 | T1543, T1554 |
| Monitoring of the tool | DE.CM-09 | AU-6, SI-4 | T1072 |
NIST SP 800-128 describes security-focused configuration management as managing and monitoring configurations to achieve adequate security, and is the reference for the baseline you rebuild to. In ISO/IEC 27001:2022 Annex A the closest controls are 8.9 (configuration management), 8.32 (change management) and 5.26 (response to information security incidents). In CIS Controls v8.1 the tool supports Control 4 (Secure Configuration of Enterprise Assets and Software). See the NIST SP 800-128 Security-Focused Configuration Management Guide.
Common failure patterns
- Restarting the controller before stopping runs: the next scheduled run reapplies the attacker's changes to every node.
- Reverting a commit but not the history: a second malicious change in a shared module or a dependency pin is missed.
- Rotating leaf certificates while keeping the CA: if the CA key was exposed, the attacker can still mint trusted node or server certificates.
- Forgetting deployed secrets: passwords and keys the tool wrote into templates and files on nodes stay valid after the controller is rebuilt.
- Trusting run reports: reports come from the agent and controller the attacker controlled; verify node state independently.
- One credential for all nodes: a single SSH key or admin account across the estate turns rotation into a big-bang change that teams postpone.
Practitioner checklist
- Image the controller and export job history, audit logs, repository history and the CA signing log.
- Disable schedules, triggers and agent runs, and block controller access to node ports.
- Lock the repository and suspend automation deploy keys.
- Revoke the tool's secret store token and cloud credentials.
- Rebuild the controller from clean media and restore only reviewed code.
- Replace the agent CA or controller SSH keys and re-enrol nodes through a verified channel.
- Rotate every secret the tool could decrypt, fetch or deploy, at its owner.
- Diff node state against reviewed desired state and rebuild unexplained nodes.
- Return to service in rings, starting in check mode, with a tested way to stop runs fleet-wide.
Standards and references
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- NIST: SP 800-128, Guide for Security-Focused Configuration Management of Information Systems
- NIST: SP 800-53 Rev. 5
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: Software Deployment Tools
- MITRE ATT&CK: Compromise Software Supply Chain
- CISA: Known Exploited Vulnerabilities Catalog
Related NHI Mgmt Group resources: Securing the configuration management tool itself · Configuration Management Tool Identity · CI/CD Pipeline Compromise Playbook · If your secrets manager is compromised