Moving to a new monitoring and observability platform is the moment an organisation is most likely to go blind without noticing, because the alerts that stop firing look exactly like a quiet week. Teams change observability platforms to cut cost, consolidate tools after a merger, adopt open instrumentation, or move metrics, logs and traces onto one backend. The work is usually framed as an engineering project, but the platform holds production alerting, service level objectives (SLOs), dashboards used during incidents, telemetry that often contains personal data and secrets, and a large set of agent credentials and API keys. This playbook is for platform, SRE and security teams running such a move. It is vendor-neutral and covers inventory, the instrumentation decision, alert and SLO parity, parallel running, the identities involved, telemetry data handling and decommissioning. Building a platform for the first time is covered in the Monitoring and observability platform rollout playbook, and hardening the platform in Securing the monitoring and observability platform itself.
Key takeaways
- Alert parity is the acceptance test. A migration is done when every alert that mattered fires on the new platform for the same conditions, not when the dashboards look the same.
- Decouple instrumentation first. Moving applications to an open protocol such as OTLP, sent through collectors you control, turns this migration and the next one into a routing change.
- Run both platforms in parallel. Dual-sending for at least one full business cycle is the only way to measure gaps in metrics, alerts and SLOs before you depend on the new platform.
- Every agent carries a credential. Ingest keys, API tokens, cloud integration roles and webhook secrets are non-human identities: reissue them scoped for the target and revoke the old set on a dated plan.
- Telemetry is sensitive data. Logs, traces and span attributes regularly contain tokens and personal data; the migration is the time to add redaction, not to copy the problem.
What an observability migration actually moves
Inventory the estate by object type before choosing tools or timelines. Most failed migrations underestimate the items in the lower half of this table.
| Object | Migration approach | Watch for |
|---|---|---|
| Agents and SDKs on hosts, containers and functions | Replace with open instrumentation or the target's agents; roll out by ring | Agents running as root or with host-wide rights; auto-instrumentation that changes application behaviour |
| Metrics and their names, labels and units | Map each metric; record renames and unit changes | Label cardinality limits on the target; counters that reset differently |
| Logs and traces | Route through collectors to the target; decide what history moves | Trace context propagation formats that differ between old and new libraries |
| Alert rules and notification routes | Rebuild by intent, then test each rule against real data | Rules with no owner, silenced rules, rules that never fired |
| SLOs and error budgets | Recreate with the same service level indicator definitions and windows | Different aggregation or window semantics changing the burn rate |
| Dashboards and runbook links | Rebuild the dashboards used in incidents; archive the rest | Runbooks and incident templates linking to old URLs |
| Synthetic checks and uptime probes | Recreate from the same locations and intervals | Probes holding test account credentials |
| Integrations: cloud accounts, paging, chat, ticketing, SOAR | New scoped credentials per integration | Cloud read roles with broad permissions copied unchanged |
| User accounts, teams and roles | Federate through the identity provider; rebuild roles from need | Local accounts and shared admin logins created during the trial |
| The platform's own audit log | Export and retain for the required period | Audit history lost when the old contract ends |
Use the old platform's own data to drive the inventory: which agents reported in the last 30 days, which alerts fired in the last 90, which dashboards were opened in the last 90 (these windows are NHI Mgmt Group recommendations). Anything with no recent use is a candidate to drop rather than migrate.
The instrumentation decision
The biggest design choice is whether applications keep sending telemetry in a platform-specific format, or move to an open standard first. The OpenTelemetry Protocol (OTLP) specification is marked stable for traces, metrics and logs, and defines default ports of 4317 for OTLP/gRPC and 4318 for OTLP/HTTP, with request paths /v1/traces, /v1/metrics and /v1/logs.
| Approach | How it works | Choose it when | Trade-off |
|---|---|---|---|
| Agent swap | Remove old agents, install the target's agents | Small estate, short timeline, few custom metrics | Re-instrumenting again at the next move; no parallel run without double agents |
| Open instrumentation plus collectors | Applications emit OTLP to collectors that export to old and new backends | Medium and large estates, multi-cloud, likely future moves | Collectors are new infrastructure to run, patch and secure |
| Bridge | Collectors receive the old agents' format and translate | Agents cannot be changed quickly (appliances, legacy hosts) | Translation gaps in fields and metric types; a temporary component that tends to become permanent |
| Big bang cutover | Switch everything on a date | Rarely justified; only for very small estates | No measured parity; rollback means reinstalling old agents |
For most organisations the collector approach is the safer default because it gives you dual-sending for free during the migration. The design of the collector layer itself is covered in the Log pipeline (collectors, forwarders, data pipeline) migration playbook, and the protocol in Observability, telemetry and SRE for security teams.
Alert and SLO parity
Alerts are where a quiet failure hurts most. Treat each one as a test case with an owner.
- Export and classify: list every alert rule with owner, severity, notification route, last fired date and linked runbook. Mark security-relevant alerts (authentication failures, privilege changes, agent stopped reporting, audit log gaps) separately so they get extra testing.
- Rewrite by intent: state in one line what each alert is meant to catch, then write the target query. Query languages and aggregation rules differ, so a literal translation can change thresholds silently.
- Back-test: replay the rule against at least 30 days of dual-sent data (NHI Mgmt Group recommendation) and compare firings with the old platform. Every difference is either explained or fixed.
- Fire drills: for critical alerts, trigger the condition deliberately in a test service and confirm the page, ticket or chat message arrives through the new route.
- Recreate SLOs: keep the same indicator definitions and compliance windows. The Google SRE workbook's multiwindow, multi-burn-rate approach, for example paging when 2% of a 30-day budget for a 99.9% SLO burns in one hour (burn rate 14.4), depends on exact window and threshold semantics, so check them on the target rather than assuming.
The SRE workbook defines alert precision, recall, detection time and reset time. Measuring all four on both platforms during the parallel run gives you an objective parity score instead of an opinion.
Where identity fits
An observability platform is a hub of non-human identities. A typical estate has an ingest key per environment, API tokens for automation and dashboards-as-code, cloud integration roles that read every account's metrics and configuration, and outbound webhook secrets for paging and chat.
- Ingest keys: issue new keys per environment or team for the target; never reuse one organisation-wide key. Store them in a secrets manager, not in agent configuration files or container images. The OpenTelemetry Collector configuration guidance notes that collector configuration can contain API tokens and TLS private keys and should be stored on an encrypted filesystem or secret store.
- Cloud integration roles: rebuild from what the target actually reads. Prefer federated, short-lived access over long-lived access keys. See the Workload Identity Federation Guide.
- Agent privileges: run agents and collectors as a dedicated non-root user where possible, as the Collector security guidance recommends, and document any component that needs elevated rights and why.
- Human access: federate users through the identity provider with SSO and MFA; remove local accounts created during evaluation.
- Revocation: keep a register of every old credential with its revocation date, and revoke after cutover rather than at contract end.
Monitoring tools are attractive targets precisely because of this access. The SolarWinds supply chain compromise showed how a widely trusted monitoring product, deployed with broad network reach, became a path into thousands of environments. See the Service Account Security Guide and the Monitoring and observability platform integration security guide.
Telemetry data handling
- Find sensitive fields before you move them: sample logs, span attributes and metric labels for tokens, session identifiers, email addresses and payment data. The OWASP Logging Cheat Sheet lists access tokens and session identifiers among data that should not be recorded directly.
- Redact at the collector: the OpenTelemetry Collector security guidance describes a redaction processor that deletes attributes not on an allow list and masks values matching blocked patterns. Apply it before data leaves your network.
- Decide on history: most teams do not migrate historical metrics; they keep the old platform read-only until its retention expires, or export to cheap storage. Write the decision down per data type, with the date the old data is deleted.
- Data residency: confirm the target's storage region matches your obligations before any personal data flows to it.
- Retention: set retention on the target from your logging policy, not from the default plan. Security-relevant logs normally belong in the log management platform under longer retention; see the Log Management and Retention Guide.
The phased playbook
- Inventory and owners: build the object inventory above, name an owner for every alert and SLO, and drop what is unused.
- Target design: tenancy and environments, role model, SSO, ingest key layout, retention, redaction rules and network paths. Bind collector receivers to specific interfaces rather than
0.0.0.0, as the Collector guidance advises. - Pilot one service end to end: instrumentation, collectors, metrics, logs, traces, alerts, SLOs, dashboards, paging and a rollback rehearsal.
- Dual-send in rings: non-production, then low-criticality production, then critical services. Expand only when the previous ring's parity metrics pass.
- Parallel run: at least one full business cycle with both platforms alerting, routed so that on-call staff are not paged twice (for example, new platform alerts go to a review channel).
- Cutover by service: switch paging for a service to the new platform once its alerts pass back-testing and fire drills. Update runbooks and incident templates in the same change.
- Freeze and revoke: stop sending to the old platform, set it read-only, revoke old keys and roles, and remove old agents.
- Decommission: export the platform audit log and any data you must retain, confirm deletion of the rest in writing, and close the contract.
Rollback: while dual-sending is in place, rollback is a routing change: point paging back to the old platform for the affected service. Keep the old platform's alerts enabled (routed to a review channel) until each service's cutover is signed off. Once old agents are removed, rollback means redeploying them, so remove agents last.
Small vs large organisations: a small team with one cloud and a few dozen services can often do an agent swap with a two-week parallel run. Large estates need the collector layer, ring-based rollout, an alert parity tracker and a dedicated owner for credential revocation.
Detection and measurement
| Metric or signal | Why it matters | Target (NHI Mgmt Group recommendation) |
|---|---|---|
| Alert parity: critical alerts back-tested and fire-drilled | Proves detection survives the move | 100% before cutover of each service |
| Hosts and services reporting to target vs source | Finds missed agents and dark services | Equal counts; every difference explained |
| Metric and log volume difference per service during dual-send | Detects dropped data or filters that are too aggressive | Within an agreed tolerance, for example 2% |
| Telemetry heartbeat gaps | A silent agent looks like a healthy service | Alert on any source silent beyond its expected interval |
| Old credentials still in use after cutover | Forgotten integrations and agents | Zero within 30 days of cutover |
| Sensitive values detected in target telemetry | Redaction gaps | Zero, with each finding fixed at the collector |
Send the target platform's own audit log (logins, role changes, key creation, alert rule changes and deletions) to your SIEM from day one. Disabling or muting monitoring is a known defence impairment technique (MITRE ATT&CK T1685, Disable or Modify Tools), and a migration creates plenty of legitimate-looking reasons to change alert rules.
Common failure patterns
- Alert rules translated literally: different aggregation or missing-data behaviour means a rule that used to fire on no data now stays silent.
- Heartbeat alerts forgotten: nobody migrates the "agent stopped reporting" alert, so hosts that never received the new agent are invisible.
- One ingest key everywhere: a single organisation-wide key is baked into images and repositories and cannot be revoked without breaking everything.
- Broad cloud roles copied over: the old integration's read-everything role is recreated for the target, and the old one is never removed.
- Old platform switched off before the parallel run proves parity: usually driven by contract dates rather than evidence; negotiate an overlap period early.
- Secrets and personal data copied with the telemetry: the new platform starts life with the same sensitive data problem, now in two places.
Framework mapping
| Activity | NIST SP 800-53 Rev 5 | NIST CSF 2.0 |
|---|---|---|
| Inventory of agents, integrations and monitored components | CM-8 | ID.AM-07 |
| Migration change control and rollback | CM-3 | PR.PS-01 |
| Monitoring coverage and alert parity | SI-4 | DE.CM-01, DE.CM-09 |
| Telemetry generation and availability | AU-2, AU-12 | PR.PS-04 |
| Agent and integration least privilege | AC-6 | PR.AA-05 |
| Encrypted telemetry transport | SC-8 | PR.DS-02 |
| Retention and disposal of old telemetry | AU-11, SI-12 | PR.DS-01 |
Practitioner checklist
- Inventory agents, metrics, alerts, SLOs, dashboards, probes, integrations and users, with owners.
- Decide agent swap, open instrumentation with collectors, or bridge, and write down why.
- Back-test every critical alert against dual-sent data and fire-drill the most important ones.
- Recreate SLOs with identical indicator definitions, windows and burn-rate thresholds.
- Issue new scoped ingest keys and integration credentials, stored in a secrets manager.
- Add redaction at the collector before telemetry reaches the target.
- Run both platforms for at least one full business cycle before cutting over paging.
- Stream the target's audit log to the SIEM and alert on alert rule deletion or muting.
- Revoke every old credential on a dated register and remove old agents last.
- Export retained data and audit history, then confirm deletion from the old platform.
Standards and references
- OpenTelemetry: OTLP Specification
- OpenTelemetry: Collector configuration best practices
- OpenTelemetry: Collector security best practices
- Google SRE Workbook: Alerting on SLOs
- OWASP: Logging Cheat Sheet
- NIST: SP 800-53 Rev 5, Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: T1685 Disable or Modify Tools
Related NHI Mgmt Group resources: Monitoring and observability platform rollout playbook · Securing the monitoring and observability platform itself · If your monitoring and observability platform is compromised · AI Agent Observability and Incident Response Guide · SIEM Migration Guide