On-Call and Paging Security: protecting alert routing, escalation and responder access

An attacker who can change who gets paged, or whether anyone gets paged at all, can switch off your detection without touching a single security tool. On-call and paging platforms sit at the end of almost every monitoring and detection pipeline, so a maintenance window, a suppression rule, an edited escalation policy or a responder's swapped phone number can silence the alert that mattered. These platforms also hold integration keys, outbound webhooks and chat connections that act as non-human identities, and a map of who responds to what. This guide treats the paging layer as security infrastructure: how routing and escalation get tampered with, how to protect responder accounts and integration credentials, how to keep paging working when the platform or the phone network fails, and how to detect tampering. It is written for SRE, platform, SOC and incident response leads, and it is vendor neutral. Platform-specific credential steps for one product are in the PagerDuty credentials guide.

Key takeaways

• Routing configuration is a detection control. Maintenance windows, suppression and routing rules, schedules and escalation policies decide whether an alert reaches a human, so changes to them need the same review as a SIEM rule change.
• Responder contact methods are authenticators in disguise. Whoever controls the phone number or app session a page goes to controls the response, so changing a contact method should be treated like rebinding an authenticator.
• Integration keys are write access to your incident queue. Inbound keys can trigger and close incidents; outbound webhooks and chat apps carry incident detail out. Inventory, scope and rotate them as non-human identities.
• Paging needs its own failure detection. A heartbeat or dead man's switch that pages when monitoring goes quiet catches both outages and deliberate silencing.
• Plan for the paging platform being unavailable or compromised. An out-of-band contact path and a tested fallback roster belong in the incident response plan.

Why the paging layer is security infrastructure

Most organisations harden their SIEM and EDR consoles and treat paging as an operations tool. That split creates a blind spot. Defence impairment techniques usually aim at the sensors (ATT&CK T1685 Disable or Modify Tools), but the same outcome is reached by leaving the sensors alone and breaking the path from alert to person. A suppressed or misrouted page leaves every upstream tool reporting healthy. The paging platform also holds sensitive data: rotation schedules naming who is reachable at 3am, service and dependency maps, and incident timelines that describe live weaknesses.

The paging layer has four parts, each with its own risks:

• Ingress: monitoring tools, SIEM and cloud alarms send events using integration keys or email addresses.
• Routing logic: event rules, deduplication, suppression, maintenance windows, service mapping and severity.
• Escalation and schedules: rotations, overrides, escalation policies and per-user notification rules.
• Egress: push, SMS, voice and email notifications, chat channels, webhooks to ticketing and status pages.

Alert routing and escalation tampering

The table maps the changes an attacker or a careless change can make to the effect on response and the control that limits each one.

Two platform behaviours show why these settings matter. PagerDuty documents that while a service is in a maintenance window its integrations are disabled and no new incidents trigger, and that users, Managers, Admins and Account Owners can create windows. The NHI Mgmt Group PagerDuty guide also notes that its audit trail report does not capture maintenance windows, disabled services or added integrations, so detection has to come from configuration exports or the API rather than the audit report alone. Check what your own platform logs before relying on its audit trail.

Responder accounts and contact methods

A responder account is privileged in an unusual way: it rarely changes infrastructure, but it receives the first signal of an attack and can acknowledge it. Takeover lets an attacker acknowledge and silence pages, read incident detail, or add their own number as a contact method.

• Single sign-on with automatic deprovisioning: require SSO for all users and provision through SCIM or directory sync, so leavers lose access and drop off rotations. Keep one or two break-glass local accounts with phishing-resistant MFA, as in the Break-Glass Account Guide.
• Treat contact method changes as authenticator binding: NIST SP 800-63B-4 says that changing a pre-registered telephone number is the binding of a new authenticator, and lists device swap, SIM change and number porting as risk indicators for the phone network. Notify the user on another channel when a contact method is added or removed, and alert the team lead when a responder's only contact method changes.
• Do not rely on SMS or voice alone: NIST treats the phone network as restricted for out-of-band authentication. For paging this is a reliability and integrity point as well as an access one: a SIM swap (ATT&CK Mobile T1451) can redirect pages. Use the mobile app push as primary with SMS or voice as a fallback, not the only path.
• Short mobile sessions on personal phones: responder apps often live on personal devices. Require device passcode, app sign-in through SSO, and session revocation when a phone is lost.
• Least privilege roles: most responders need to acknowledge, resolve and add notes, not edit services, routing or users. Separate the platform admin role from day-to-day on-call.
• Production access stays separate: being on call should trigger a time-bound elevation request, not standing production rights. See Production Access for SREs and On-Call Engineers.

Integration keys, webhooks and chat apps

Paging platforms accumulate machine credentials faster than most teams track them. Every monitoring tool that sends events holds a key, every outbound webhook holds a URL and often a secret, and chat and ticketing connections hold OAuth grants.

How two platforms document this (listed alphabetically):

• Atlassian Opsgenie: distinguishes team integration keys, which act only on that team's alerts and team-based configuration, from global integration keys, which can be used for all API requests. Keys created under API Key Management carry Read, Create/Update, Delete and Configuration Access rights and are created by account owners and global admins. Grant only the rights each use needs. See the Atlassian Cloud Security Guide.
• PagerDuty: separates account REST API keys, user tokens, OAuth app credentials and Events API integration keys; integration keys can trigger, acknowledge and resolve incidents on their service. Create REST keys read-only where possible. See the PagerDuty credentials guide.

Keys for paging tend to end up in monitoring configuration files, infrastructure-as-code state and CI variables (T1552.001 Credentials In Files). Scan those locations and handle a found key with the Leaked Credential Response Playbook. Chat-driven incident actions are covered in ChatOps Security.

Resilience: when paging itself fails

Paging is a dependency of incident response, so its failure modes belong in the continuity plan (NIST SP 800-53 CP-8 Telecommunications Services). The aim is that silence is never mistaken for health.

• Heartbeat or dead man's switch: each critical monitoring source sends a regular check-in; if it stops, the paging platform raises an alert. This catches crashed collectors, expired keys and deliberate silencing alike.
• Synthetic end-to-end pages: trigger a test event weekly from each critical source to a test service and confirm a human acknowledgement within the target time.
• Out-of-band roster: keep a current offline contact list for incident leads and a fallback bridge, so the team can mobilise if the platform, SSO or the corporate chat is down or compromised.
• Independent path for security alerts: for the highest-severity detections, consider a second notification path that does not share the same routing rules.
• Supplier risk: include the paging provider in third-party risk reviews (CSF 2.0 GV.SC-07), including its status communications and data residency.

Sustainable on-call load is also a security property. The Google SRE book suggests no more than two incidents per 12-hour shift and tuning noisy alerts towards a 1:1 alert to incident ratio. Responders drowning in pages acknowledge without looking, which is exactly the condition an attacker generating fake incidents wants. See the Alert Triage Guide for the triage side.

Choosing controls by size

• Small teams (one rotation, few services): SSO, MFA on admin accounts, two escalation levels, a heartbeat on the main monitoring source and a printed contact list cover most risk. Review integrations twice a year.
• Mid-sized organisations: add configuration-as-code for routing and escalation, tagged security services with restricted maintenance window rights, and SIEM alerts on configuration changes.
• Large or regulated organisations: separate platform administration from on-call, require change records for routing changes on security services, run quarterly paging failover tests, and keep configuration exports as evidence for incident response audits.

How to implement

1. Inventory: export users, roles, schedules, escalation policies, services, integrations, API keys, webhooks and chat apps. Tag services that carry security detections.
2. Fix identity first: enforce SSO, connect directory provisioning, remove leavers from rotations, and reduce admin role holders.
3. Scope and rotate credentials: replace shared or account-wide keys with per-source keys, move them into a secrets manager and set owners. Rotate in a test service first, then production, keeping the old key until events are confirmed on the new one.
4. Baseline routing as code: capture escalation, routing and suppression configuration in version control with peer review. Start in report mode (diff and alert) before blocking manual changes.
5. Add heartbeats and synthetic pages: pilot on one critical source, then extend. Roll back by disabling the heartbeat alert, not the source.
6. Wire detection: stream platform audit logs and configuration diffs into the SIEM with the alerts below.
7. Exercise: include "paging platform unavailable" and "routing tampered" injects in incident response tests (NIST SP 800-53 IR-3).

Detection and measurement

Alert on: maintenance windows on security services, routing or suppression rule changes, escalation policy edits, schedule overrides to users outside the team, contact method changes, new API keys or webhooks, disabled integrations, bulk acknowledges or resolves, and incidents resolved within seconds by an integration rather than a human.

• Time to acknowledge for security services: median and 90th percentile against target.
• Synthetic page success rate: share of weekly test pages acknowledged within target, aiming for 100 per cent.
• Heartbeat coverage: share of critical monitoring sources with an active heartbeat.
• Unreviewed routing changes: configuration changes on security services without a matching review, target zero.
• Credential hygiene: integration keys and API keys with a named owner and rotation date, and stale keys removed per quarter.
• Pages per shift: tracked against a sustainable ceiling to catch alert flooding.

Common failure patterns

• Maintenance windows used as a mute button: windows are opened for noisy services and never closed, so a real intrusion on that service pages nobody.
• Admin rights for everyone on call: every responder can edit routing and integrations, so any single account takeover can silence alerting.
• Leavers still on the rotation: manual user management leaves departed staff on schedules and as escalation targets.
• One account-wide key in every monitoring tool: a single leaked key can forge or close incidents everywhere and is hard to rotate.
• Incident data flowing to public channels: webhooks and chat integrations post hostnames and responder details where too many people, or guests, can see them.
• No test of the paging path itself: teams discover during an outage that the heartbeat was never configured or the fallback list is a year old.

Framework mapping

Relevant ATT&CK techniques: T1685 Disable or Modify Tools (impairing detection), T1078 Valid Accounts, T1098 Account Manipulation, T1552.001 Credentials In Files and, for mobile, T1451 SIM Card Swap.

Practitioner checklist

• SSO and directory provisioning are enforced, with leavers removed from rotations automatically.
• Platform admin rights are held by a small named group separate from day-to-day on-call.
• Security services are tagged and only admins can open maintenance windows on them, always with an end time.
• Every security service has at least two escalation levels.
• Routing, suppression and escalation configuration is version controlled and diffed daily.
• Contact method changes notify the user on another channel; push is primary and SMS or voice a fallback.
• Integration keys are per source, owned, stored in a secrets manager and rotated on exposure.
• Outbound webhooks and chat apps are allow-listed and reviewed quarterly.
• Heartbeats cover every critical monitoring source, and synthetic pages run weekly.
• An offline contact list and fallback bridge are in the incident response plan and tested.

Standards and references

• NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• NIST: SP 800-63B-4, Authentication and Authenticator Management
• Google: Site Reliability Engineering, Being On-Call
• Atlassian: Opsgenie API Key Management
• PagerDuty: Maintenance Windows
• MITRE ATT&CK: T1685 Disable or Modify Tools

Related NHI Mgmt Group resources: PagerDuty credentials · IT Operations Security Maturity Model · Alert Triage Guide · Break-Glass Account Guide · ChatOps Security