A compromised monitoring and observability platform gives an attacker a privileged agent on almost every host, read access to every cloud account it watches, and the ability to decide what your operations and security teams see. These platforms are trusted by design: their agents run with root or SYSTEM rights, their cloud integrations hold cross-account roles, and their dashboards, traces and logs often contain tokens and personal data. This guide is a response playbook for a suspected or confirmed compromise of the platform itself, whether it is self-hosted or delivered as a service. It covers how the incident shows up, triage, containment in priority order, credential rotation, rebuild, evidence and notification. It is written for incident responders, SRE and platform teams, and security operations leads, and it stays vendor-neutral.
Key takeaways
- Treat the platform as a privileged identity hub. Every agent key, cloud integration role, API key, webhook and stored check credential is in scope from the first hour, not only the console logins.
- Contain the control plane before the agents. Freeze remote configuration, custom check deployment and integrations first, because those are the paths from the platform to your hosts.
- Assume monitored hosts are suspect until proven clean. The SUNBURST response directive told agencies to treat every host monitored by the compromised platform as compromised and to rebuild the platform from trusted media.
- Rotate in dependency order. Admin identities and SSO trust first, then cloud integration roles and keys, then agent and ingest keys, then credentials stored for checks, and include credentials used in the past.
- Restore visibility from a second source. While the platform is untrusted, rely on native cloud audit logs, the SIEM and host logs that never passed through it.
Why this compromise is different
Most incidents start with a user or a server. This one starts with the system you use to watch users and servers. The platform aggregates three things attackers want: privileged execution on hosts (agents and remote check runners), broad read access to infrastructure (cloud, database and network integrations), and a curated view of telemetry that hides or reveals activity. The table shows what an attacker gains from each component and how urgently it needs action.
| Component | Typical credential or privilege | What an attacker gains | Priority |
|---|---|---|---|
| Admin console and API | SSO users, local break-glass users, admin API keys, application keys | Change monitors, mute alerts, push agent configuration, read all telemetry | 1 |
| Agent fleet | Agent or ingest key, agent running as root or SYSTEM, remote configuration channel | Code execution on every monitored host if remote commands or custom checks are enabled (ATT&CK T1072 Software Deployment Tools) | 1 |
| Cloud integrations | Cross-account role with external ID, access keys, service principal secrets, service account keys | Read-only, and sometimes wider, view of every connected account (T1078.004 Cloud Accounts) | 1 |
| Database, network and middleware checks | Stored database logins, SNMP community strings, SSH keys, JMX credentials | Direct access to databases and network devices | 2 |
| Synthetic and browser tests | Test user passwords, MFA seeds, API tokens in scripts | Valid application logins, sometimes privileged test tenants | 2 |
| Notification integrations | Chat and paging webhooks, ticketing tokens | Suppress or spoof alerts, phish responders through trusted channels | 2 |
| Telemetry content | Logs, traces and dashboards that captured tokens, headers or personal data | Secrets in clear text (T1552.001 Credentials In Files) and a data breach in its own right | 3 |
Self-hosted and service-delivered platforms fail differently. For a self-hosted platform you own the servers, database and agent update channel, so a server compromise can reach the agent fleet. For a service, the likely routes are stolen admin sessions, leaked API or application keys, or an incident at the provider (ATT&CK T1199 Trusted Relationship). See the Securing the monitoring and observability platform itself guide for the preventive baseline.
How the compromise shows up
Few teams detect this from the platform alone, because the attacker can edit the platform's own alerting. Expect the first signal to come from somewhere else.
| Signal | Where to look | What it may mean |
|---|---|---|
| Monitors muted, deleted or thresholds raised outside change windows | Platform audit trail, configuration-as-code repository diff | Defence impairment (ATT&CK TA0112, T1685 Disable or Modify Tools) |
| New API or application keys, new admin users, SSO settings changed | Platform audit trail, identity provider sign-in and admin logs | Persistence (T1098 Account Manipulation) |
| Integration role used from unexpected IPs or calling write APIs | Cloud provider audit logs for the integration principal | Stolen integration credential (T1078.004) |
| New custom checks, scripts or agent integrations pushed to hosts | Agent configuration directories, host process creation and file integrity logs | Execution through the agent (T1072) |
| Agent binary or package hash differs from the published release | Package manager logs, file integrity monitoring, software inventory | Tampered agent or update channel (T1554, T1195.002) |
| Unusual bulk export of logs, traces or dashboards | Platform audit trail, API usage metrics | Collection of secrets and personal data |
| Provider security notice or a public exploit for your platform version | Provider status and trust pages, CISA KEV | Third-party or vulnerability-driven compromise |
Monitoring consoles are a recurring entry in the CISA Known Exploited Vulnerabilities Catalog, for example CVE-2021-39226 (authentication bypass), CVE-2022-23131 (authentication bypass) and CVE-2024-6670 (SQL injection) in widely deployed self-hosted monitoring products. An internet-exposed self-hosted console on an affected version is a reason to open this playbook even without other signals.
Triage questions for the first hour
- Which plane is affected? Console and API, agent fleet, integrations, the provider, or a combination. The answer decides how much of the estate is in scope.
- Can the platform push code or configuration to hosts? Check whether remote configuration, remote command execution, custom checks or auto-update are enabled, and on how many hosts.
- Which cloud accounts, databases and network devices does it authenticate to? Pull the integration list and the stored credential list before anything is changed, and export them to the case file.
- Is the audit trail intact and independent? If audit events live only inside the platform, assume they may be altered and look for an exported copy in the SIEM or cloud storage.
- What sensitive data does the telemetry hold? Search for authorisation headers, bearer tokens, connection strings and personal data in logs and traces over the retention window.
- What visibility do we keep if we switch it off? List the alternative sources (cloud audit logs, SIEM, EDR) before containment removes monitoring.
Record each answer with time and owner. NIST SP 800-61 Rev. 3 frames this as the triage and validation outcome RS.MA-02 and the analysis outcome RS.AN-03 of CSF 2.0.
Containment by priority
- Preserve first, then cut. Export the audit trail, user list, API key list, integration list, monitor definitions and agent configuration. For self-hosted servers, take memory and disk images before powering down where you have the capability, as ED 21-01 required for affected monitoring servers.
- Freeze the control plane. Disable remote configuration, remote commands and custom check deployment globally. Pin agent versions and stop auto-update until the update channel is verified.
- Lock admin access. Restrict console and API access to a named response group, revoke active sessions, disable local accounts except one monitored break-glass account, and enforce phishing-resistant MFA through SSO.
- Disable or narrow cloud integrations. Remove the integration role trust or detach its policies in each connected account. Disabling breaks dashboards but stops the most valuable access path; do it account by account if a full outage is unacceptable.
- Isolate self-hosted servers. Block traffic between the platform servers and the internet, and restrict inbound agent traffic to what you still need. ED 21-01 also required blocking traffic between affected hosts and destinations outside the enterprise.
- Protect notification paths. Rotate paging and chat webhooks and move incident communication to an out-of-band channel the attacker cannot read.
Decision: shut down the platform completely or keep it running in a restricted state. Shut it down when the agent channel or server is confirmed compromised. Keep it in restricted, read-only mode when the compromise is limited to a leaked read key and the platform is your main source of production telemetry.
Small and large organisations: a small team using a service-delivered platform can often revoke everything and reconnect integrations in a day. A large estate with many accounts and thousands of agents should contain by plane, then rotate integrations account by account and agent keys ring by ring.
Credential rotation sequence
Rotate in an order that does not hand the attacker the new secret. The SUNBURST directive required agencies to reset all credentials used by or stored in the monitoring platform, including SNMP strings and accounts used previously, and to remove all inbound trust relationships to the rebuilt system.
| Order | What to rotate | How | Verify |
|---|---|---|---|
| 1 | Admin users, SSO configuration, admin API and application keys | Re-issue from SSO, delete unused keys, scope new keys to one function | Key inventory matches the approved list; no key older than the incident start |
| 2 | Cloud integration roles and keys | Replace access keys with role assumption, regenerate external IDs, rotate service principal secrets and service account keys | Cloud audit logs show only the new principal from expected sources |
| 3 | Agent and ingest keys | Issue new keys, roll out to the fleet through your configuration management tool, revoke old keys | Old keys rejected; agent count back to baseline |
| 4 | Stored check credentials (databases, SNMP, SSH, JMX, synthetic test users) | Rotate on the target system, not only in the platform; move to SNMPv3 and least-privilege monitoring users | Failed logins with old values appear and are investigated |
| 5 | Secrets found in telemetry | Rotate every token or password found in logs and traces, then purge or restrict that data | Search for the same patterns returns nothing new |
Rotation is where many responses fail: one missed key keeps the attacker in. The Trivy supply chain attack shows how an incomplete rotation let attackers back into a trusted security tool. See the Credential Rotation at Scale guide.
Eradication and rebuild
- Self-hosted platform: rebuild servers from a fresh operating system and installation media verified against published hashes, as ED 21-01 required for the affected monitoring infrastructure. Do not restore the old application database wholesale; re-import monitor and dashboard definitions from version control after review.
- Separate the tiers: host the application, web front end and database on separate hosts, an ED 21-01 condition for continued use of the affected platform, and keep them off domain-joined admin workstations.
- Agents: reinstall agents from verified packages, remove any custom checks or scripts not in version control, and compare agent configuration directories against a known-good baseline.
- Monitored hosts: hunt on hosts where the agent could execute code, starting with crown-jewel systems. Rebuild hosts where unexplained changes appear. See the Network Rebuild After Compromise Playbook.
- Service-delivered platform: work with the provider on their findings, delete and recreate the organisation or tenant if admin persistence cannot be ruled out, and re-establish integrations with new credentials only.
- Recovery gate: resume normal use only when key inventories are clean, integrations use new principals, and a hunt over monitored hosts has found no further activity. CSF 2.0 RC.RP-03 asks that restoration assets are verified before use.
Evidence to preserve and notification triggers
| Evidence | Why it matters |
|---|---|
| Platform audit trail export (users, keys, monitor changes, exports) | Timeline of control-plane actions and data access |
| Cloud audit logs for every integration principal | Scope of access into connected accounts |
| Agent configuration directories and custom check scripts | Proof of, or exclusion of, execution through agents |
| Server memory and disk images (self-hosted) | Persistence and root cause analysis |
| Identity provider sign-in logs for platform users | Session theft, MFA fatigue or token replay |
| Results of searches for secrets and personal data in telemetry | Basis for rotation and breach notification decisions |
Notification depends on what the telemetry contained and where you operate. Personal data in logs or traces can make this a reportable personal data breach, and essential or regulated entities may have incident reporting duties. Bring legal and privacy teams in as soon as the telemetry search finds personal data. If the platform is a service, check the provider contract for mutual notification terms. The Sumo Logic breach is a reminder that a provider incident can require customers to rotate their own keys even when their data was not confirmed as accessed.
Where identity fits
Nearly everything you rotate in this playbook is a non-human identity: integration roles, service principals, agent keys, API keys and monitoring service accounts. If they are not in an inventory with owners, the response stalls on discovery. Monitoring service accounts on databases and network devices are often over-privileged because "read everything" was easier than scoping. Replace long-lived keys with role assumption or workload identity federation where the platform supports it. See the Service Account Security Guide and the API Key Management Guide.
Detection and measurement
Send the platform's audit trail to the SIEM in near real time so the attacker cannot edit the only copy, and alert on monitor deletion, mute rules, new keys and integration changes. Watch integration principals in cloud audit logs for any write call. The following measures show whether the response worked and whether the platform is safer afterwards.
- Time to freeze the control plane: minutes from declaration to remote configuration and custom checks being disabled.
- Rotation coverage: percentage of inventoried platform credentials rotated, target 100 percent, with the remainder named.
- Old credential use: count of authentication attempts with revoked keys after rotation, each one investigated.
- Integration privilege: number of integration principals with write permissions, target zero unless documented.
- Audit trail independence: percentage of platform audit events present in the SIEM within five minutes.
- Secrets in telemetry: weekly count of tokens or passwords detected in logs and traces, trending to zero.
Framework mapping
| Activity | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | MITRE ATT&CK |
|---|---|---|---|
| Triage and scoping | RS.MA-02, RS.AN-03 | IR-4, IR-5 | T1199, T1078.004 |
| Containment of control plane and agents | RS.MI-01 | IR-4, CM-5, AC-6 | T1072, T1685 |
| Credential rotation | RS.MI-02 | IA-5, SC-12 | T1552.001, T1528 |
| Rebuild and integrity checks | RC.RP-01, RC.RP-03 | SI-7, CM-2 | T1554, T1195.002 |
| Independent audit trail | PR.PS-04, DE.CM-09 | AU-6, AU-9(2), SI-4 | TA0112 |
| Provider risk | GV.SC-07 | SA-9, SR-3 | T1199 |
In ISO/IEC 27001:2022 Annex A the closest controls are 5.26 (response to information security incidents), 8.15 (logging) and 8.16 (monitoring activities). In CIS Controls v8.1 the platform sits under Control 8 (Audit Log Management) and Control 17 (Incident Response Management).
Common failure patterns
- Rotating the console password and nothing else: integration roles, agent keys and stored check credentials stay valid and the attacker keeps access to every connected system.
- Forgetting past credentials: ED 21-01 called out credentials used previously by the platform. Old SNMP strings and retired database users often still work.
- Restoring the old database: a full restore brings back rogue API keys, users and custom checks along with the dashboards.
- Trusting the platform's own audit trail: if the only copy is inside the compromised system, the timeline cannot be relied on.
- Ignoring secrets in telemetry: tokens captured in logs and traces were readable by the attacker and stay valid after the platform is rebuilt.
- Scoping only the platform, not what it watched: the SUNBURST supply chain compromise showed a monitoring platform used as the first step towards identity infrastructure and cloud email.
Practitioner checklist
- Export audit trail, users, keys, integrations and agent configuration before changing anything.
- Disable remote configuration, remote commands, custom checks and agent auto-update.
- Restrict console and API access to the response group and revoke all sessions.
- Detach or narrow every cloud integration role and record which accounts lose visibility.
- Rotate credentials in dependency order, including credentials used in the past.
- Search logs and traces for secrets and personal data, rotate what you find and involve privacy counsel.
- Rebuild self-hosted servers from verified media and re-import definitions from version control.
- Hunt on hosts where the agent could execute code, starting with identity and crown-jewel systems.
- Stream the platform audit trail to the SIEM and alert on key, user, monitor and integration changes.
Standards and references
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- CISA: Emergency Directive 21-01, Mitigate SolarWinds Orion Code Compromise
- CISA and ASD ACSC: Best Practices for Event Logging and Threat Detection
- NIST: SP 800-53 Rev. 5
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: Software Deployment Tools
- CISA: Known Exploited Vulnerabilities Catalog
Related NHI Mgmt Group resources: Securing the monitoring and observability platform itself · If your SIEM is compromised · If your log pipeline (collectors, forwarders, data pipeline) is compromised · Cloud Credential Compromise Playbook