When a security or IT management tool is compromised, the attacker inherits the tool's trust: its agents, its integrations, its stored credentials and the alerts it was meant to raise. NHI Mgmt Group publishes dedicated compromise playbooks for many tool classes, from SIEM and log pipelines to secrets managers and patch tools. This page is the front door for everything else: a short, vendor-neutral triage and routing playbook for any security or IT tool that has no dedicated page, and for the first hour of any tool incident before you know which class-specific playbook applies. It is for incident managers, SOC leads, IAM teams and the platform owners who run these tools. Supplier-side incidents (where the vendor itself is breached) are handled with the Third-Party Incident Response Playbook; this page covers the steps inside your own estate.
Key takeaways
- Classify the tool by what it can do, not what it is called. Four powers decide the response: whether it executes code on endpoints, whether it holds credentials, whether it can change configuration elsewhere, and whether it feeds your detection.
- Route early. If a dedicated playbook exists for the class, switch to it within the first hour; use this page only to bridge the gap and for tools with no page.
- Revoke the tool's identities before you touch the tool. Integration tokens, service accounts and API keys live in other systems; cutting them there works even if the attacker still owns the console.
- Assume your telemetry is lying. A compromised security tool may have suppressed alerts or rules, so verify from an independent source before declaring scope.
- Rebuild, do not patch in place. Patching a compromised system does not remove an attacker who is already inside; restore from known good media, then reconnect with narrower scope.
Classify the tool in five minutes
Most tool compromises follow the same shape, whatever the product. What changes is the blast radius, and that is set by the tool's capabilities. Answer these four questions from the tool's architecture documentation or your configuration management records, then use the table to set severity and pick the containment order.
| Capability | Typical examples (classes) | What the attacker gains | Severity driver |
|---|---|---|---|
| Executes code or scripts on managed hosts | Remote management, patching, configuration management, endpoint agents, software distribution | Mass execution across the estate (ATT&CK T1072 Software Deployment Tools) | Number of hosts with the agent, and whether servers or domain controllers are among them |
| Holds credentials for other systems | Vaults, discovery scanners, backup, monitoring, integration and automation platforms | Every stored secret, plus the systems those secrets reach (T1552 Unsecured Credentials) | Privilege of the stored accounts; whether domain or cloud admin keys are present |
| Changes configuration or identity elsewhere | Identity governance, provisioning connectors, network management, ticket-driven automation | New accounts, group changes and policy changes made through a trusted channel (T1098 Account Manipulation) | Write scope of its connectors |
| Feeds or controls detection | SIEM, log pipelines, scanners, posture tools, email filtering | Blind spots: disabled rules, dropped logs, suppressed findings (T1685 Disable or Modify Tools) | Share of your detection that depends on it |
A tool that scores on two or more rows is high severity by default. A tool that scores on none (for example a read-only dashboard with no stored credentials) can usually be handled as an ordinary application incident. In practice, most enterprise security tools score on at least two.
Route to the right playbook
Check this table before going further. If your tool matches a class, open its playbook now and use the rest of this page only for steps the class playbook leaves to you, such as coordination and notification.
No match? Typical examples are a niche discovery tool, an internal admin portal, a licence or inventory agent, a network monitoring console, or a home-grown automation server. Continue with the steps below.
Triage: the first hour
- Open the incident and name an owner. Use the severity from the classification table. Start a decisions log with time, decision, evidence and approver.
- Locate the start point. Your instance (stolen admin login, leaked API key, exploited vulnerability on a self-hosted server), its agents or connectors, the tool's update channel (T1195.002 Compromise Software Supply Chain), or the provider that runs it for you (T1199 Trusted Relationship). The start point decides who leads: you, or the supplier route.
- List the tool's identities. Every service account, API key, OAuth grant, certificate and agent credential the tool uses, and every account other systems use to reach it. This list drives containment.
- List what it stores. Credentials, scan results, configuration backups, session recordings, inventories. Anything it stores is now in scope.
- Check for active misuse. Look in the systems the tool talks to, not only in the tool: new accounts or keys created by its service identity, jobs or scripts pushed to agents, unusual data exports and logins by the tool's accounts from new sources (T1078 Valid Accounts).
- Find an independent view. If the tool feeds detection, identify a second source (firewall or proxy logs, cloud provider audit logs, identity provider sign-in logs) to confirm what it reports.
Decision point T1, ongoing or historic: recent activity from unknown sources means containment starts now, in parallel with investigation. Historic-only evidence allows a planned rebuild, but still preserve evidence first.
Containment by priority
Preserve evidence before each destructive step (see the evidence table below), then work in this order. The order matters: the tool's power lives in other systems, so cutting it there is fast, reversible and does not depend on the compromised console.
- Stop remote execution. If the tool pushes code or scripts, pause job scheduling and deployment queues, and block agent-to-server traffic at the network layer for the server if pausing cannot be trusted. Do not uninstall agents yet: they are evidence.
- Cut its identities in the target systems. Disable its service accounts in the directory, revoke its API keys and OAuth grants in each connected platform, and remove its network allow-list entries. Record the previous state of each for rollback.
- Isolate the tool server or tenant. Restrict management interfaces to an incident jump host; disable all admin accounts except named responders; revoke active sessions. For a self-hosted appliance, segment it rather than power it off, so memory and running state survive for forensics.
- Freeze its write paths. Pause provisioning connectors, change workflows and configuration pushes that run through it, so nothing it sends is applied while scope is unknown.
- Cover the blind spot. If it was a detection tool, raise sensitivity on the independent sources from triage step 6 and assign an analyst to watch them until the replacement is live.
Decision point T2, suspend or keep running: some tools (backup, endpoint protection, email filtering) protect you while they run. If suspending removes a protection, keep the function running in a restricted mode with its credentials rotated, or stand up a temporary replacement, and record who accepted the risk.
Rollback: every containment step above is a revocation or a block in another system. Keep a list of each change with its previous state, so normal operation can be restored deliberately, one connection at a time.
Credential rotation
Rotate in order of what an attacker can do with each credential, not in order of convenience. The Secrets Rotation Runbook Template gives the per-secret procedure; this table sets the sequence for a tool incident.
| Order | Credential | Action | Proof it worked |
|---|---|---|---|
| 1 | Privileged accounts stored in the tool (domain admin, cloud admin, root, database admin) | Reset at source; for directory accounts follow your domain recovery procedure if the domain may be affected | Old credential fails; no use of it after the reset time |
| 2 | The tool's own service accounts and API keys in other systems | Replace with new, narrower identities; delete the old ones after the rebuild | Only the new identity appears in target audit logs |
| 3 | Agent enrolment secrets and server certificates | Reissue; re-enrol agents to the rebuilt server only | Agents reject the old server; the old server has no enrolled agents |
| 4 | Integration secrets (webhooks, ticketing, chat, SIEM forwarding) | Rotate on both ends | Signature checks fail for the old secret |
| 5 | Human admin accounts and their MFA | Reset passwords, re-register MFA, revoke sessions and personal tokens | No sessions older than the reset time |
Discovery and scanning tools often hold credentials that nobody remembers adding. Compare the tool's credential store export with your secrets inventory, and rotate the union, not only the ones you knew about.
Eradication and rebuild
- Rebuild from known good media. Australian Signals Directorate guidance on edge devices states that applying a patch will not remedy a device that has already been compromised. The same logic applies to any tool server: rebuild the host or tenant from trusted installation media or a fresh instance, apply current vendor updates, then restore configuration from a backup taken before the earliest compromise date.
- Review configuration before you restore it. Restored rules, exclusions, jobs, scripts and connector settings can carry the attacker's changes. Diff against a pre-incident export and remove anything unexplained.
- Clean what the tool pushed. For execution tools, list every job, script and package deployed in the window, and check the receiving hosts. Hosts that ran attacker jobs follow your endpoint isolation and rebuild route: see How to Isolate a Compromised Endpoint.
- Undo identity changes. Accounts, group memberships, keys and grants created or changed by the tool's identity in the window must be reviewed and reversed.
- Restore detection. Re-enable rules and log sources that were disabled, and backfill missed log data from the independent sources where possible.
Evidence to preserve and notification
| Evidence | Where | Why it matters |
|---|---|---|
| Tool audit log and admin login history | The tool itself, exported before isolation | Shows who changed what; may be tampered, so compare with other sources |
| Job, task and deployment history | Tool database or scheduler | Lists what was executed, where and when |
| Sign-in and API logs for the tool's identities | Identity provider, cloud audit logs, connected SaaS | Independent record of what the tool's credentials did |
| Server image and memory | Self-hosted server or appliance | Persistence and attacker tooling; capture before rebuild |
| Network flows to and from the tool | Firewall, proxy, flow logs | Exfiltration and command traffic |
Extend retention on every source now. ASD's practitioner guidance on edge devices recommends sending logs to a central system in real time and making sure the device itself cannot modify or delete them, which is the property you need for a tool incident too.
Decision point T3, notification: Legal and the privacy lead decide using confirmed facts. Under GDPR Article 33(1) a controller notifies the supervisory authority without undue delay and, where feasible, not later than 72 hours after becoming aware of a notifiable personal data breach; under Article 33(2) a processor notifies the controller without undue delay. Entities in scope of the NIS2 Directive send an early warning for a significant incident within 24 hours of becoming aware of it and an incident notification within 72 hours, under Article 23(4). A tool that stored personal data, or whose compromise disrupted an essential service, can trigger either. If the tool is supplied or run by a third party, tell the supplier and follow the supplier route in parallel.
Decision guidance
| Situation | Choose | Trade-off |
|---|---|---|
| Tool has a dedicated playbook | Switch to it after triage | None: the class page has the specific containment points |
| Compromise confined to one admin account, no execution or credential access seen | Contain the account, rotate the tool's identities, keep the instance after review | Faster recovery; accept residual risk if audit logs were incomplete |
| Server-level compromise, tampered update or unknown persistence | Full rebuild and re-enrolment | Longer outage; the only way to be confident |
| Tool protects you while running | Restricted mode or temporary replacement | Some exposure remains; record the risk owner |
| Small organisation, tool run by a provider | Revoke the provider's access paths, then follow the supplier route | You depend on the provider's evidence; keep your own logs |
Large organisations with many instances of the same tool should contain all instances that share credentials, update channels or an admin directory, even where only one shows signs of compromise.
Common failure patterns
- Trusting the tool's own audit log. An attacker with admin rights can alter or delete it. Confirm with sign-in and network logs from other systems.
- Revoking only the credentials you knew about. Scanners and automation tools accumulate stored accounts; rotate from the tool's own export, not from memory.
- Patching and declaring victory. A patch closes the entry point but leaves accounts, keys and implants in place.
- Uninstalling agents first. Agents and their logs are evidence, and removing them can leave hosts without protection; stop the server side first.
- Restoring the attacker's configuration. Backups taken after the compromise began bring back malicious jobs, exclusions or connector changes.
- Reconnecting everything at once. Reconnect one integration at a time, read-only first, so a missed persistence point shows up quickly.
Detection and measurement
Watch the rebuilt tool and its identities closely for at least 30 days. Alert on: the tool's service identities used from any source other than the rebuilt server; new admin accounts or API keys in the tool; jobs pushed outside change windows; rules, exclusions or log sources disabled; and agents checking in to any server other than the rebuilt one.
- Time to route: minutes from declaration to the decision on which playbook leads.
- Time to revoke: hours from declaration until every listed tool identity is disabled in its target system.
- Rotation coverage: percentage of credentials in the tool's export rotated and verified.
- Re-enrolment rate: percentage of agents enrolled to the rebuilt server, with stragglers investigated.
- Playbook coverage: percentage of security and IT tools in the inventory with a named playbook (dedicated or this page) and an identity list ready before an incident.
Framework mapping
| Framework | Reference | Where it applies |
|---|---|---|
| NIST CSF 2.0 | RS.MI-01 Incidents are contained; RS.MI-02 Incidents are eradicated | Containment and rebuild sections |
| NIST CSF 2.0 | RS.MA-01 response plan executed with relevant third parties; RC.RP-01 recovery executed | Supplier route, reconnection |
| NIST SP 800-53 Rev. 5 | IR-4 Incident Handling; IR-6 Incident Reporting; IA-5 Authenticator Management; AC-2 Account Management | Triage, notification, rotation, identity clean-up |
| CIS Controls v8.1 | Control 17 Incident Response and Management | Owner, roles, playbook coverage |
| MITRE ATT&CK | T1072, T1078, T1098, T1195.002, T1199, T1552, T1685 | Classification table and hunting |
Practitioner checklist
- Classify the tool against the four capabilities and set severity.
- Check the routing table and switch to a dedicated playbook if one exists.
- List every identity the tool uses and every credential it stores.
- Preserve the tool's audit log, job history, server image and external logs before revoking.
- Pause remote execution and write paths, then disable the tool's identities in target systems.
- Rotate stored privileged credentials first, then the tool's own identities, agents and integrations.
- Rebuild from known good media and restore a reviewed, pre-compromise configuration.
- Reconnect one integration at a time with narrower scope, and monitor for 30 days.
- Record notification decisions under GDPR, NIS2 and contracts, even when nobody is notified.
Standards and references
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- Australian Signals Directorate: Mitigation strategies for edge devices, practitioner guidance
- CISA: Partners release guidance on edge devices
- EUR-Lex: Regulation (EU) 2016/679 (GDPR), Article 33
- EUR-Lex: Directive (EU) 2022/2555 (NIS2), Article 23
- CIS: Control 17, Incident Response and Management
- MITRE ATT&CK: T1072 Software Deployment Tools
Related NHI Mgmt Group resources: Third-Party Incident Response Playbook · Incident Response Plan Template · Secrets Rotation Runbook Template · Network Rebuild After Compromise Playbook