AI Agents in DevOps and SRE: Safe Production Access

AI agents are being given the same production access as on-call engineers, but without the judgement, the change process or the audit trail that access was designed around. AIOps triage bots, SRE agents that restart workloads, and coding agents that run terraform apply all need some route into production, and the incidents so far show what happens when that route is a broad, standing credential. This guide sets out an operating model for agents in DevOps and SRE work: how to scope cloud and Kubernetes roles, how to put changes through approval and dry runs, how break-glass works when an agent is involved, and what to log so you can tell what the agent did and on whose behalf. It is written for platform, SRE and cloud IAM teams. The incidents themselves are analysed in the AI agents that destroyed data guide; this page is the build guide.

Key takeaways

Read-only by default, write by exception. Most SRE agent value (triage, correlation, runbook lookup, drafting a fix) needs read access only. Separate the read identity from any write identity, and make write access short-lived and approved.
Agents propose, pipelines apply. Route agent changes through the same pull request, plan and approval path as human changes, so the change system, not the prompt, decides what reaches production.
Dry runs need the same permission as the real action. Kubernetes server-side dry run is authorised exactly like the real request, so "dry-run only" is not a permission boundary on its own.
Break-glass is for humans. An agent should never hold or be able to request emergency access by itself; it can page a human who does.
Attribute every action to an agent and a person. Session identifiers such as AWS SourceIdentity, plus agent-side logs of prompts and tool calls, are what make an agent incident investigable.

What SRE and DevOps agents actually do

"AI agent in production" covers very different levels of risk. Classify each agent use case by the highest action it can take, then give it the access that class needs and nothing more.

Use case	Typical actions	Access needed	Risk class
Alert triage and incident summary	Read metrics, logs, traces, recent deploys; post to the incident channel	Read-only observability and deploy history	Low
Diagnosis on a live system	Describe pods, read events, query cloud resource state	Read-only Kubernetes and cloud API, no secrets	Low to medium (data exposure)
Runbook automation	Restart a deployment, scale a service, fail over a replica	Narrow write on named resources, time-bound	Medium
Infrastructure change authoring	Write Terraform or Helm changes, open a pull request	Repository write on a branch; no cloud write	Medium (supply chain)
Infrastructure change execution	Run `terraform apply`, `kubectl apply`, database migrations	Cloud and cluster write, often broad	High
Destructive or data operations	Delete volumes, drop tables, rotate keys, change IAM	Admin-level	Critical: keep away from agents

The lesson from the Replit AI agent database deletion in July 2025 is the one to design around: the user had declared a code freeze, the agent deleted the production database anyway, and the user later said there was no way to enforce the freeze technically. Replit's response was platform controls, including separation of development and production databases and a planning-only mode that cannot change the codebase. Instructions in a prompt are not controls; access boundaries are. OWASP's LLM06 Excessive Agency entry describes the same root causes: excessive functionality, excessive permissions and excessive autonomy.

Scoping cloud roles for agents

Give each agent its own workload identity, never a shared engineer's credentials or a long-lived access key in an environment file. Then cap what that identity can ever do, independently of whatever policy someone attaches to it later. Platforms are listed alphabetically.

Control	AWS	Google Cloud	Microsoft Azure / Entra
Hard ceiling on permissions	Permissions boundary: effective permissions are the intersection of identity policy and boundary, and of any SCP and session policy; an explicit deny in any wins	IAM Conditions on role grants, applied through Privileged Access Manager entitlements	Scope role assignments to a resource group or resource rather than a subscription
Time-bound elevation	Short session duration on the assumed role; session policies to narrow a single session	Privileged Access Manager grants with a maximum duration per entitlement, revoked automatically at expiry	Privileged Identity Management: time-bound, just-in-time activation with approval and justification
Approval	Through your change pipeline or access broker	Optional approvals on entitlements; multi-level approvals and agent identity approvers in Preview	PIM approval to activate privileged roles
Attribution	`SourceIdentity` set at `AssumeRole`, persists through role chaining and appears in CloudTrail as `userIdentity.sessionContext.sourceIdentity`	PAM events in Cloud Audit Logs	PIM audit history and activation notifications

Separate read and write identities: The agent's default identity reads; a second, write-capable role is assumed only for an approved change and only for the change window. Two identities make it obvious in logs when the agent crossed from looking to acting.
Bound role creation: If an agent can create IAM roles or policies at all, AWS documents a pattern where creation is allowed only when the new entity carries a named permissions boundary (condition key iam:PermissionsBoundary) and deleting the boundary is denied. Without this, "can create roles" is "can become admin".
Use SourceIdentity for the human: AWS states that the value persists for the whole role session and cannot be changed, and that a different value during role chaining is denied. Set it to the requesting engineer or ticket, and require it in trust policies with sts:SourceIdentity conditions.
No credentials within reach: An agent that can read a repository, a CI variable or a home directory can find tokens there (ATT&CK T1552.001 Credentials In Files). Agents should get credentials only through federation at run time. See the Terraform and IaC Pipeline Credential Security Guide and the JIT Access Guide.

Kubernetes RBAC for AI agents

The Kubernetes RBAC good practices page lists the permissions that look harmless but are not. For an agent's ServiceAccount these are the lines to check first.

Permission	Why it matters for an agent	Rule
Wildcards (`*`) on verbs or resources	Kubernetes is extensible, so wildcards silently cover new resources and CRDs	Name verbs and resources explicitly
`cluster-admin`	Full control; the docs say not to use it except where specifically needed	Never bind to an agent
`get`, `list`, `watch` on Secrets	Exposes every credential in scope, and secrets end up in model context and logs	Diagnosis agents do not need Secrets; exclude them
`create` on Pods or workloads	Implicitly grants access to Secrets, ConfigMaps and volumes that can be mounted, and the API rights of any ServiceAccount in the namespace	Only in a dedicated namespace with the Restricted Pod Security Standard
`get` on `nodes/proxy`	Reaches the kubelet API; the docs stress this is not read-only	Never for agents
`escalate`, `bind`, `impersonate`	Each bypasses Kubernetes' built-in protection against privilege escalation	Never for agents
`create` on `serviceaccounts/token`	Mints tokens for other ServiceAccounts	Never for agents

A workable pattern is a cluster-wide read role that excludes Secrets for diagnosis, plus a namespaced Role allowing only patch on named Deployments (for restarts and scaling) granted through your elevation process when a runbook needs it. The docs note that boundaries within a namespace are weak, so put workloads with different trust levels in different namespaces. See the Kubernetes RBAC Guide.

Change approval and dry runs

The safest SRE agent never applies anything itself: it writes a change, proves what it will do, and a pipeline applies it after approval. Where agents do execute, make the preview step mandatory and machine-checked.

Terraform saved plans: HashiCorp documents terraform plan -out=FILE as the two-step workflow intended for automation: the saved plan is reviewed, then applied exactly with terraform apply FILE. A plan without -out is only speculative. Have the agent produce the saved plan; have the pipeline apply it after a human approves.
Machine-check the plan: -detailed-exitcode returns 0 for no changes, 1 for error and 2 for changes present. terraform show -json gives machine-readable output so a policy check can block any plan containing delete or replace actions on protected resource types.
prevent_destroy: Terraform's prevent_destroy lifecycle rule rejects plans that would destroy the resource. HashiCorp notes it does not stop destruction if the resource's configuration is removed, so pair it with a review rule on deleted resource blocks.
Kubernetes server-side dry run: The API accepts ?dryRun=All (kubectl --dry-run=server), runs admission, and guarantees nothing is persisted. Authorisation for a dry run is identical to the real request, so an agent allowed to dry-run a patch is allowed to patch. Use dry run as evidence in the change record, not as a restriction.
Approval in the change system: Classify agent changes the way you classify human ones: standard (pre-approved runbook action), normal (reviewed pull request) and emergency (human-led). See the Change Management Security guide.

Decision guidance: a small team without a mature change pipeline should keep agents to read-only diagnosis plus pull requests, and apply changes by hand. A large platform team with GitOps can let agents execute pre-approved standard changes (restart, scale within limits, roll back to the previous release) through the pipeline, and should route everything else through review. Neither should let an agent execute IAM changes, deletions of stateful resources or database schema drops.

Break-glass, incidents and the on-call agent

Incidents are when pressure to give the agent more access peaks. Decide in advance.

The agent pages, a human elevates. The on-call agent can recommend an action and pre-fill the elevation request; a named human activates break-glass access, using the documented procedure in the Break-Glass Account Guide.
No agent in the break-glass path. Break-glass credentials must not be readable by any agent identity, and break-glass roles should not trust agent principals.
Freeze means freeze. Implement freezes as a technical control: remove the agent's write role binding, or block agent identities at the pipeline. A message in the chat is not a freeze.
Kill switch. Be able to revoke the agent's sessions and disable its identity in one action. See the AI Agent Kill Switch and Containment Guide.
Recovery outside the blast radius. Keep backups in a separate account or project the agent cannot reach. Deleting data and its backups maps to ATT&CK T1485 Data Destruction and T1490 Inhibit System Recovery, whether an attacker or an agent does it.

Implementation roadmap

Inventory. List every agent with any infrastructure access, its identity, where its credentials come from and the highest action it can take.
Remove static credentials. Replace keys in files and CI variables with federated, short-lived credentials per agent.
Split read and write. Create a read-only identity per agent and move write rights to a separate role behind elevation.
Add ceilings. Apply permissions boundaries or equivalent scoping; strip Secrets, wildcard and escalation verbs from Kubernetes roles.
Route changes through the pipeline. Saved plans, policy checks for destructive actions, human approval for anything beyond standard changes.
Pilot in a non-production ring. Run the agent against staging for several incident cycles; review every action before granting the same scope in production.
Roll back. If an agent misbehaves, revoke its write role first (diagnosis can continue on read-only), then investigate with the attribution logs.

Where identity fits

Each SRE agent is a non-human identity with an owner, a purpose and an expiry, and should be governed like one: registered, reviewed and deprovisioned when the use case ends. The human it acts for is a second identity that must appear in the logs. Agents that hold MCP or tool credentials for observability, ticketing and chat systems add more machine credentials to inventory. See the AI Agent Sandboxing and Code Execution Guide for the execution side.

Detection and measurement

Cloud audit logs: All write and delete calls by agent identities, with SourceIdentity or equivalent; any agent call to IAM, KMS key deletion or backup deletion APIs.
Kubernetes audit logs: Agent ServiceAccount requests for Secrets, pod exec, nodes/proxy and RBAC objects (all should be zero).
Agent-side logs: Prompt, plan, tool call and result for each action, linked by a session ID to the cloud audit record.
Pipeline: Plans blocked by policy, applies without an approved saved plan.

Metrics:

Percentage of agent identities with no standing write access to production.
Percentage of agent changes that went through a saved plan and approval.
Number of agent actions in audit logs that cannot be attributed to a requesting human or ticket (target zero).
Count of static credentials reachable by agents (target zero).
Mean time to revoke an agent's write access during a drill.

Framework mapping

Control	NIST SP 800-53 Rev. 5	NIST CSF 2.0	ATT&CK / OWASP
Per-agent identities, owners, deprovisioning	AC-2 Account Management	PR.AA-05	T1078.004 Cloud Accounts
Read-only defaults, ceilings, no escalation verbs	AC-6 Least Privilege	PR.AA-05	OWASP LLM06 Excessive Agency
Agent proposes, human approves	AC-5 Separation of Duties; CM-3 Configuration Change Control	PR.PS-01	OWASP LLM06 (excessive autonomy)
Change rights only through the pipeline	CM-5 Access Restrictions for Change	PR.PS-01	OWASP LLM06 (excessive functionality)
Attribution of agent actions	AU-12 Audit Record Generation	DE.CM-03	T1078.004 (detecting misuse of cloud accounts)
Backups outside the agent's reach	CP-9 System Backup	PR.DS-11	T1485, T1490
No credentials in files	AC-6 Least Privilege	PR.AA-05	T1552.001 Credentials In Files

Common failure patterns

Running the agent as the engineer. Agents launched in an engineer's terminal inherit their cloud session and kubeconfig, including admin roles, and logs show the engineer did it.
Freeze by instruction. As the Replit case showed, telling the agent not to change anything does not stop it. Remove the permission.
Trusting "dry run only". Granting an agent patch rights "for dry runs" grants patch rights.
Diagnosis roles that read Secrets. Broad read roles copied from human SRE groups often include Secrets, so credentials flow into model context, transcripts and third-party logs.
Backups in the same account. An identity that can delete the database can usually delete its snapshots too. The AI agents that destroyed data guide shows this happening.
No owner. Pilot agents left running with production roles after the pilot ends become orphaned privileged identities.

Practitioner checklist

Give every agent its own federated workload identity; remove keys from files and CI variables.
Default each agent to read-only and keep write rights in a separate, time-bound, approved role.
Apply permissions boundaries or equivalent scoping, and bound any role creation to a named boundary.
Remove Secrets access, wildcards, cluster-admin, nodes/proxy, escalate, bind, impersonate and token creation from agent ServiceAccounts.
Route agent infrastructure changes through saved plans, policy checks for destructive actions and human approval.
Keep break-glass credentials and roles out of reach of all agent identities.
Implement code freezes and kill switches as permission changes, and drill them.
Store backups in an account or project no agent identity can reach.
Log agent prompts and tool calls with a session ID that links to cloud audit records and the requesting human.

AI Agents in DevOps and SRE: Production Access Without Disasters