Production Access for SREs and On-Call Engineers

On-call engineers need production access in minutes, at any hour, and that pressure is why so many organisations leave SREs with standing admin rights. A pager going off at 3am is the worst moment to wait for an approver, so teams grant permanent access "just in case", and that access sits there for an attacker every other hour of the year. This guide shows how to give site reliability engineers and on-call responders fast, time-bound production access tied to incidents, how to get shell and cluster access without long-lived keys, and how to keep session evidence an auditor and a post-incident review can use. It is for SRE and platform leads, PAM owners and security engineers. The general theory of just-in-time access is in the JIT Access Guide, emergency accounts are in the Break-Glass Account Guide, and agents doing SRE work are in AI Agents in DevOps and SRE. This page covers the human on-call engineer.

Key takeaways

Default to read, elevate to write. On-call engineers keep standing read-only access to dashboards, logs and cluster state; write and exec rights are activated per incident.
Tie elevation to the pager, not to a person's goodwill. Being the current on-call responder with an open incident is the condition for fast, pre-approved elevation; everything else goes through peer approval.
No long-lived SSH keys or kubeconfigs. Use short-lived SSH certificates or a cloud session service, and identity-provider-issued cluster credentials, so access ends when the elevation ends.
Record the session, not just the sign-in. Sign-in logs say who got in; session logs, cluster audit logs and cloud API logs say what they did, which is what a post-incident review needs.
Clean up after the incident. Anything created during the incident (new keys, role bindings, debug pods, firewall openings) is listed and removed or ticketed before the incident is closed.

Why production access for humans is its own problem

The JIT pattern assumes someone can approve a request in reasonable time. On-call breaks that assumption three ways. The requester is often the only engineer awake. The work is urgent, so any delay is measured in customer impact. And the actions taken are exactly the high-risk ones: restarting services, editing configuration, running database fixes, opening a shell on a host. The result is predictable: if elevation is slow, engineers request the maximum duration, share an admin session, or keep a personal backdoor. The design goal is an elevation path that is faster than the workaround and still leaves evidence.

Access tier	Typical rights	How it is granted (our recommendation)	Maximum duration (our recommendation)
Observe	Dashboards, logs, traces, read-only cloud and cluster views	Standing, for the SRE group	Standing, reviewed quarterly
Operate	Restart, scale, roll back a deployment, run approved runbook actions	Self-activated by the current on-call engineer with an incident ID	1 to 4 hours, renewable
Change	Edit configuration, IAM, network rules, exec into containers, shell on hosts	Activated with an incident ID; peer approval when not on call	1 hour, renewable
Data	Query or modify production databases holding customer data	Peer approval always, with stated justification	Per task, under 1 hour
Emergency	Directory or cloud root recovery when the normal path has failed	Break-glass procedure only	Until recovery, then credentials reset

Splitting "Data" from "Change" matters: infrastructure repair rarely needs to read customer records, and keeping those separate is the cheapest way to make a stolen on-call session less valuable.

Designing incident-linked elevation

Eligibility from the rota: sync the on-call schedule into the identity system so that only the engineers currently on call, plus named incident commanders, are eligible for self-activation of Operate and Change roles.
Incident ID as justification: require an incident or change ticket number on every activation. The JIT tool records it, and reviewers can join it to the incident timeline later.
Fresh, phishing-resistant authentication on activation: step up with a FIDO2 security key or platform passkey at activation, not only at morning sign-in. The Uber Breach 2022 began with a contractor's stolen password and repeated push approvals until one was accepted; elevation protected by push MFA inherits that weakness. See the MFA Fatigue and Bypass Guide.
Peer approval outside the incident path: when the requester is not on call, or wants Data access, a second engineer approves. Make sure the approver pool has someone awake in every time zone you operate in.
Notifications to the incident channel: post every activation to the incident channel and the security operations queue, so others see who holds elevated access right now.
No elevation by editing role bindings: engineers must not grant themselves access by editing IAM directly. The JIT system is the only path, and direct IAM edits by humans alert.

Cloud provider elevation features

Providers are listed alphabetically. Each row names the provider's own feature, from its documentation as of November 2026; confirm current behaviour in the provider's docs.

Provider	Time-bound elevation	Host access without inbound SSH or RDP	Where the evidence lands
Amazon Web Services	IAM Identity Center temporary elevated access, which AWS documents as request, approve and track for a specific task in a specified time, delivered through validated partner integrations	AWS Systems Manager Session Manager, controlled by IAM policies, with no inbound ports, bastion hosts or SSH keys to manage	AWS CloudTrail for session API calls; Session Manager session logs to Amazon S3 or CloudWatch Logs; EventBridge rules for session start and stop
Google Cloud	Privileged Access Manager: entitlements define eligible principals, roles, maximum grant duration, justification and approvers; roles are removed automatically when the grant ends	OS Login, linking Linux accounts to Google identities through IAM roles `roles/compute.osLogin` and `roles/compute.osAdminLogin`, with `enable-oslogin-2fa` to require two-factor authentication	Cloud Audit Logs for entitlements, grant requests and approvals, with Pub/Sub notification through a logging sink
Microsoft Azure	Microsoft Entra Privileged Identity Management: just-in-time and time-bound role activation with approval, MFA on activation, justification and notifications, for Entra ID and Azure resource roles	Azure Bastion, providing RDP and SSH over TLS on port 443 to virtual machines by private IP, with Entra ID authentication for native client connections	PIM audit history; Bastion session recording

Availability notes: AWS: temporary elevated access is delivered through AWS Security Competency partner solutions integrated with IAM Identity Center; Session Manager does not log session content for port forwarding or SSH sessions, so plan other evidence for those. Google Cloud: check which Privileged Access Manager features are generally available in your organisation, and note that PAM manages time-bound bindings in IAM allow policies, so leave those bindings to PAM. Microsoft Azure: check the Bastion SKU documentation for which tiers include session recording and native client Entra ID authentication.

Shell, cluster and database access without standing keys

Hosts

Where a cloud session service is not used, issue short-lived OpenSSH user certificates from an SSH certificate authority instead of distributing public keys. ssh-keygen signs a user key with options that matter for on-call: -V sets the validity interval (for example one hour), -n sets the principals the certificate is valid for, -I sets a key identity that sshd logs on use, and -O adds restrictions such as source-address, no-port-forwarding or force-command. Put the incident ID in the key identity so host logs join back to the incident. Alert on any new entry in authorized_keys on production hosts (ATT&CK T1098.004 SSH Authorized Keys), because a key added during an incident outlives the certificate.

Kubernetes

Identity-provider credentials: engineers authenticate to clusters with short-lived tokens from the identity provider, never a shared admin kubeconfig. Bind the Operate and Change roles to groups that the JIT system populates.
Exec is Change, not Operate: pods/exec and pods/attach give a shell inside workloads that often hold secrets. Keep them out of standing roles.
Audit policy: the API server audit policy sets levels None, Metadata, Request and RequestResponse. Log at least Metadata for secrets (never Request or RequestResponse, which would copy secret values into logs) and RequestResponse for role bindings and exec, and ship events through the log or webhook backend to storage the cluster's admins cannot alter. Each event records the user, any impersonated user and the verb.

For incident handling on clusters, see the Kubernetes Cluster Compromise Playbook.

Production databases

Route human database access through a broker or bastion that issues per-session credentials and records queries. Separate read-only diagnostic roles from write roles, and require a second person for writes to tables holding customer data. The application's own database credential is a non-human identity and must never be borrowed for a manual fix: see the Secrets Management Guide.

Session evidence and post-incident review

Question in the review	Evidence source
Who held elevated access during the incident, and why?	JIT activation log with incident ID and approver
What did they run on hosts?	Session service logs or session recording; `sshd` logs with certificate key identity
What did they change in the cloud?	Cloud control plane audit logs filtered to the engineer's session
What did they change in clusters?	Kubernetes audit events by user and verb
What did they read in production data?	Database broker query log
Was anything left behind?	Diff of IAM bindings, keys, firewall rules and workloads before and after the incident window

Add an "access hygiene" step to the incident close-out template: list every credential, binding and network change made under elevation, and either revert it or open a ticket with an owner and date. See the Incident Response Guide for the wider process.

Implementation

Inventory (weeks 1 to 2): list every human with standing production write, exec, shell or database access, and every shared admin credential or kubeconfig. Map each to Observe, Operate, Change, Data or Emergency.
Build the read-only baseline (weeks 2 to 4): give the SRE group good standing read access first. If engineers cannot see, they will ask to keep write.
Pilot JIT with one team (weeks 4 to 8): wire the rota to eligibility, require incident IDs and phishing-resistant step-up, and keep standing roles in place but unused. Measure activation time.
Move host and cluster access (weeks 6 to 12): introduce SSH certificates or a session service and identity-provider cluster credentials. Keep the old keys exported and disabled, not deleted, for rollback during the pilot.
Remove standing write (after two clean on-call rotations): remove the standing roles. Rollback is re-adding the group binding, which you document and test once.
Run a game day: practise an incident at night with JIT only, including the break-glass path, and fix whatever slowed responders down.

Small teams: with five engineers, a shared rota and one cloud provider, the provider's native elevation feature plus SSH certificates is usually enough. Larger organisations: centralise eligibility and evidence across clouds and clusters, and give incident commanders a view of all active elevations.

Detection and measurement

Alert on: elevation outside an open incident or change; activation by someone not on the rota; direct IAM edits by human users; new SSH authorized keys or long-validity certificates; exec into pods holding secrets; database writes outside the broker; break-glass use. Valid cloud accounts used from unusual locations map to ATT&CK T1078.004 Cloud Accounts, and new bindings or keys to T1098 Account Manipulation.

Metric	Target (our recommendation)
Engineers with standing production write, exec or shell access	0 outside the emergency set
Median time from page to active elevation	Under 2 minutes
Activations carrying a valid incident or change ID	100%
Incidents closed with an access hygiene step completed	100%
Long-lived SSH keys and shared kubeconfigs in use	0
Elevated sessions with retrievable session evidence	100% of Change and Data sessions

Framework mapping

Framework	Item	How this guide applies it
NIST SP 800-53 Rev. 5	AC-2(6) Dynamic Privilege Management	Incident-linked elevation and automatic removal
NIST SP 800-53 Rev. 5	AC-6(2) Non-privileged Access for Nonsecurity Functions; AC-6(5) Privileged Accounts	Read-only baseline, privileged roles only on activation
NIST SP 800-53 Rev. 5	AC-17 Remote Access; AU-14 Session Audit	Session services, SSH certificates and session evidence
NIST CSF 2.0	PR.AA-05; PR.PS-04; DE.CM-03	Least privilege, log generation, monitoring of personnel activity
MITRE ATT&CK v20	T1078.004; T1098; T1098.004; T1021.004 SSH	Behaviours the detections above look for

Common failure patterns

JIT slower than the workaround: approvals that need a manager at night lead to shared sessions and personal admin accounts. Pre-approve on-call activation instead.
Elevation without step-up: a session cookie from the morning is enough to activate admin rights hours later.
Exec in a standing role: read-only on paper, but pods/exec gives a shell next to the workload's secrets.
Leftovers after the incident: a debug key, an open security group or a temporary role binding is never removed.
Admin credentials in runbooks and scripts: a runbook that embeds a password turns one foothold into privileged access, as the attacker claimed in the Uber case.
Evidence in the blast radius: session logs stored where the elevated engineer, or an attacker with their session, can delete them.

Practitioner checklist

Every SRE has standing read-only access and no standing write, exec, shell or data access.
On-call rota drives eligibility for self-activated Operate and Change roles.
Every activation needs an incident or change ID and phishing-resistant step-up.
Data access to customer records is a separate role with peer approval.
Host access uses short-lived SSH certificates or a cloud session service; no long-lived keys.
Cluster access uses identity-provider tokens; exec only in Change roles; audit policy covers role bindings and exec.
Session evidence stored outside the reach of the engineers and the systems they administer.
Incident close-out includes an access hygiene step.
Break-glass path tested in a game day alongside JIT.

Production Access for SREs and On-Call Engineers: fast elevation at 3am with evidence afterwards