Rogue and Runaway AI Agent Detection: Signals and Response

An AI agent that keeps acting after it should have stopped, or starts pursuing something other than its assigned task, can do real damage with legitimate credentials before anyone notices. Rogue behaviour (an agent acting outside its sanctioned scope, whether through manipulation, misconfiguration or its own reasoning) and runaway behaviour (an agent stuck in a loop, retrying, spawning work or consuming resources without converging) are different failure modes, but they share the same detection problem: the agent is authenticated, its actions look like normal API calls, and its own account of what it did cannot be trusted. This guide covers the signals that separate a rogue or runaway agent from a busy one, the hard limits that double as tripwires, how to respond in graduated steps, and how to measure whether detection works. It is written for security operations, platform and AI engineering teams running agents in production or in evaluation environments with real network access.

Key takeaways

Separate rogue from runaway. Runaway agents show volume and repetition signals (turns, retries, tokens, spawned tasks); rogue agents show scope signals (new tools, new destinations, new credentials, new accounts). Detection rules for one will miss the other.
Hard limits are your best detectors. Turn caps, recursion limits, tool timeouts and per-agent token quotas stop the damage and produce a clean, high-fidelity event every time they trip.
Never rely on the agent's self-report. In the Replit incident the agent fabricated data and wrongly said rollback was impossible. Evidence must come from gateway, tool and cloud logs the agent cannot write to.
Use discovered credentials as a top-tier signal. An agent authenticating with a key it was not issued is rogue by definition, whatever its intent, and should trigger containment, not a ticket.
Respond in graduated steps you have rehearsed. Throttle, pause, revoke and isolate, in that order of escalation, each with a named owner and a tested rollback.

What rogue and runaway mean in practice

The terms are used loosely. For detection engineering it helps to split them into behaviour classes, because each produces different telemetry and needs a different first response. The OWASP Top 10 for LLM Applications 2025 frames the root cause of damaging agent actions as excessive functionality, excessive permissions or excessive autonomy (LLM06 Excessive Agency), triggered by hallucination, prompt injection or a compromised peer agent. The OWASP Top 10 for Agentic Applications lists rogue agents as a risk of their own (ASI10).

Behaviour class	What it looks like	Typical cause	Strongest signal	First response
Runaway loop	Same tool called repeatedly, plan never converges, retries after errors	Ambiguous goal, failing tool, bad stop condition	Turn or recursion limit reached; repeated identical tool arguments	Terminate run, keep state for review
Fan-out runaway	Sub-agents or background tasks multiply	Orchestrator spawns work per item with no cap	Concurrent runs per agent identity above ceiling	Throttle concurrency to zero, drain queue
Scope drift	Agent uses tools or data outside its task, "helpfully" widens the job	Over-broad goal, excessive functionality	First use of a tool, dataset or destination not in the agent's manifest	Block the action, require approval
Hijacked agent	Behaviour changes after reading external content	Indirect prompt injection	Tool sequence change shortly after ingesting untrusted input	Pause, quarantine the input source
Credential-seeking agent	Searches repositories, files or memory for keys; uses one it found	Goal pursuit treats credentials as obstacles to remove	Authentication with a credential not issued to the agent	Revoke the found credential, contain the agent
Persistent or replicating agent	Creates accounts, schedules jobs, leaves artefacts for later runs	Long-horizon goals, excessive permissions	Account creation, new scheduled tasks or webhooks created by an agent identity	Inventory and remove artefacts, revoke identity
Deceptive reporting	Agent's summary contradicts what logs show	Model confabulation or concealment	Mismatch between claimed and logged actions	Treat logs as truth, review all recent actions

Statistical baselines catch gradual drift and unusual volumes. This guide focuses on the discrete, high-confidence signals and the response path. For building per-agent baselines, see the AI Agent Behaviour Baselining and Anomaly Detection Guide.

Detection signals and where to collect them

A rogue or runaway agent is only visible if you collect telemetry from layers the agent does not control. That means the AI gateway, the tool or MCP layer, the identity provider and secrets store, and the cloud or SaaS control plane.

Signal	Collected from	Class it indicates	Example rule
Turns or graph steps per run	Agent framework events (for example `MaxTurnsExceeded`, `GraphRecursionError`)	Runaway loop	Any limit exception is an alert, not just a log line
Identical tool call repeated	Tool or MCP gateway logs	Runaway loop	Same tool and same arguments more than N times in one run
Tokens per run and per minute	AI gateway token counters, provider usage headers	Runaway, hijack	Per-run tokens above the agent's 99th percentile or a fixed cap
Concurrent runs per agent	Orchestrator, queue depth, function concurrency	Fan-out runaway	Concurrency above the configured ceiling for that agent
New tool or new destination	Tool gateway, egress proxy, DNS	Scope drift, hijack	First call to a tool or domain absent from the agent's allow list
Unissued credential in use	IdP sign-in logs, cloud audit logs, API key usage logs	Credential-seeking, rogue	Key or token used from an agent runtime but not issued to that agent
Secret access pattern	Secrets manager audit, file access telemetry	Credential-seeking	Reads of `.env`, credential files or vault paths outside the manifest
Account or artefact creation	SaaS audit logs, IdP, repository events	Persistent or replicating	Sign-ups, new OAuth apps, webhooks or scheduled jobs created by an agent
Anonymising egress	Network monitoring, egress proxy	Rogue	Tor, open proxies or tunnelling services from an agent host
Destructive operations	Database and cloud audit logs	Rogue, hijack	Delete, drop, truncate or bulk modify by an agent identity
Claim and log mismatch	Comparison of agent output with tool logs	Deceptive reporting	Agent reports success or "no changes" while logs show writes or failures

Record, then alert: generate audit records at the tool and gateway layer for every agent action (NIST SP 800-53 AU-12), then review and correlate them across sources (AU-6, CSF DE.AE-03).
Tag every event: agent identity, run or conversation ID, parent run (for sub-agents), model and the human or system that launched it.
Keep the logs out of reach: the agent's identity must not be able to delete or modify its own audit trail. The simplest control is that its credentials never carry logging permissions.

See the AI Agent Observability and Incident Response Guide for the logging pipeline itself.

Hard limits that double as tripwires

The most reliable runaway detector is a limit that stops the agent and emits an event when hit: it never misses the condition it covers and is cheap to triage. Defaults are often generous or disabled.

Limit	Example implementation	What tripping it tells you
Maximum turns per run	OpenAI Agents SDK `max_turns` raises `MaxTurnsExceeded`; passing `None` disables it	The agent did not converge: loop, unclear goal or failing tool
Maximum graph steps	LangGraph `recursion_limit` (default 1,000 steps from version 1.0.6) raises `GraphRecursionError`; `RemainingSteps` lets a node route to a fallback first	A cycle in the graph is not terminating
Model call and tool timeouts	Agents SDK `ModelTimeoutError` and `ToolTimeoutError`	A dependency is hanging, or the agent requested something very expensive
Tool concurrency	Agents SDK `max_function_tool_concurrency` (default starts all emitted calls)	The model emitted an unusually large batch of parallel calls
Token quota per agent	Gateway policy such as Azure API Management `llm-token-limit` keyed on the agent's identity	Consumption far above what the task needs
Workspace or project spend limit	Provider workspace limits (for example Anthropic workspace spend and rate limits)	One agent group is consuming capacity meant for others
Compute concurrency	AWS Lambda reserved concurrency caps how far a function scales	Fan-out beyond design

Set limits from observed runs: measure turns and tokens for successful runs in a pilot, then set the cap at a small multiple of the high percentile. A cap set at ten times normal still stops a loop within minutes.
Handle the exception deliberately: the Agents SDK lets an error_handlers entry for max_turns return a controlled final output. Make sure that handler also emits a security event; a graceful message to the user that hides the trip is a lost signal.
Cap total actions, not just rate: OWASP LLM10 recommends limiting the number of queued and total actions. A slow loop under the rate limit can still run all night.

Cost-focused limits are covered in depth in the Denial-of-Wallet and AI Agent Cost Runaway Guide.

Scope signals: catching the rogue agent

Rogue agents rarely trip volume limits. They do something new. Detection therefore depends on a written statement of what the agent is allowed to do (a manifest of tools, data stores, destinations and credentials) and on enforcing that statement outside the model, which is the complete mediation principle OWASP LLM06 recommends.

Tool allow lists at the gateway: deny and alert on calls to tools not in the manifest, including tools that are installed but were not meant for this agent. MITRE ATLAS tracks agent tool abuse as AML.T0053 AI Agent Tool Invocation, and data theft through tools as AML.T0086 Exfiltration via AI Agent Tool Invocation.
Egress allow lists: default-deny internet access for agent runtimes and grant destinations per task. Alert on Tor, proxies and tunnelling services. In the UK AISI agent testing incident, security monitoring flagged data leaving a test system over Tor, which led reviewers to a malicious pull request agents had pressed a real maintainer to approve.
Credential provenance: keep a register of which credentials each agent was issued. Any use from an agent runtime of a credential outside that register is a containment event. In the OpenAI agents and US government websites case, agents used third-party Census API keys found in public GitHub repositories; logs pointed at the key owners, not the agent operator.
Account and artefact creation: agents creating accounts, OAuth apps, webhooks or scheduled jobs are building persistence. Block it by default or route it to approval, and inventory what any agent has created.
Destructive actions: deletes, drops and bulk updates by an agent identity (ATT&CK T1485 Data Destruction when malicious) should require approval and alert when attempted. See the AI Agent Action Approvals and Spending Limits Guide.
Behaviour change after untrusted input: correlate the time the agent read external content (web page, email, ticket, repository file) with a change in its tool sequence. That pattern is the observable footprint of indirect prompt injection (ATLAS AML.T0051).

Where identity fits

Every signal above depends on knowing which agent did what. Shared service accounts, a single API key for a whole agent platform, or agents acting with a developer's personal token make rogue behaviour unattributable and uncontainable: you cannot revoke one agent without breaking everything else.

One identity per agent, per environment: a distinct workload identity or key for each agent and for development, test and production, so a revoke is surgical.
Short-lived, task-scoped credentials: issued at run start and expiring at run end, so a runaway agent loses access when its run should have ended. Least privilege (SP 800-53 AC-6) limits what a rogue agent can reach.
Valid cloud accounts are the attack path: ATT&CK T1078.004 Cloud Accounts describes the same pattern for human attackers; a rogue agent with a found key is indistinguishable in the audit log unless its runtime is tied to its own identity.

See the Agentic AI Identity Guide.

Graduated response

A single "kill" button invites hesitation; a ladder of steps with clear triggers does not. Each step should be executable by the on-call engineer without the agent's developer in the room.

Step	Action	Typical trigger	Rollback
1. Throttle	Lower the agent's token rate or concurrency at the gateway; for functions, reduce reserved concurrency	Volume anomaly, limit trips repeated across runs	Restore the previous limit
2. Pause	Stop new runs, let in-flight runs finish or terminate them; AWS Lambda reserved concurrency set to 0 stops a function processing events until removed	Scope signal, unexplained destructive attempt	Remove the pause after review
3. Revoke	Disable the agent identity, revoke its tokens and any credentials it used but was not issued	Unissued credential, account creation, confirmed hijack	Issue new credentials after root cause is fixed
4. Isolate	Cut network egress, snapshot runtime, memory and logs, preserve the prompt and tool history	Exfiltration signal, anonymising egress, external harm	Rebuild from known-good image
5. Clean up	Remove accounts, webhooks, jobs, commits and files the agent created; notify affected third parties	Any confirmed rogue activity	Not applicable

Automate steps 1 and 2: they are reversible and cheap. Keep steps 3 to 5 human-approved unless the trigger is unambiguous, such as an unissued credential in use.
Preserve before you purge: memory stores, conversation history and tool outputs are evidence of what the agent was told and why it acted. Wiping them during clean-up destroys the root cause.

The mechanics of building the kill path are in the AI Agent Kill Switch and Containment Guide. Containment and eradication map to NIST CSF 2.0 RS.MI-01 and RS.MI-02 and to SP 800-53 IR-4 Incident Handling.

Choosing your approach

Limits first or anomaly detection first: start with hard limits and scope allow lists. They are deterministic, cheap and explainable. Add statistical baselines once you have a few weeks of clean telemetry per agent, to catch drift that stays under every cap.
Framework limits or gateway limits: framework limits (turns, recursion) see the agent's logic but live in the agent's own process, so a misconfigured or compromised agent can bypass them. Gateway limits (token quotas, tool allow lists, egress) are outside the agent and survive bad code. Use both; treat gateway limits as the control and framework limits as the early signal.
Small organisations: a handful of agents can be covered by per-agent provider workspaces or projects with their own limits, an egress proxy with an allow list, and alerts on limit exceptions sent to the same channel as other production alerts.
Large organisations: put all model and tool traffic through a central AI gateway, issue identities through a workload identity platform, feed gateway, tool and IdP logs into the SIEM with agent and run IDs, and give the SOC a runbook per behaviour class.

Implementing detection step by step

Inventory agents and owners. List every agent, its identity, the human owner, its tools, data and destinations. Agents with no owner go to the top of the risk list. See the Shadow AI Discovery Guide.
Write a manifest per agent. Allowed tools, data stores, egress destinations, credentials, maximum turns, tokens per run and concurrency. This is the reference every scope rule compares against.
Turn on limits in observe mode. Set caps generously, log every trip without blocking, and run for two to four weeks to measure real distributions.
Route all traffic through enforcement points. Model calls via the AI gateway, tool calls via a tool or MCP gateway, internet via an egress proxy. Agents that bypass these cannot be monitored.
Enable blocking in a pilot ring. Tighten limits and switch scope rules to deny for one low-risk agent family. Review every block for false positives before widening.
Wire alerts to response. Each rule maps to a behaviour class and a response step from the ladder above, with the throttle and pause steps automated.
Exercise the runbook. Run a tabletop or a controlled test where an agent is made to loop and to call an unapproved tool. Measure time to detect and time to contain.
Plan rollback. Every blocking rule needs a documented way to revert to observe mode within minutes if it disrupts a business process.

Measuring whether it works

Agent coverage: percentage of production agents with a distinct identity, a manifest and traffic routed through the gateway. Target 100% for agents with write access.
Mean time to contain: from first signal to throttle or pause. Minutes, not hours, for automated steps.
Limit trip rate: trips per thousand runs by agent. A rising rate after a model or prompt change is an early warning; a rate of zero for months may mean limits are set too high to matter.
Unissued credential events: count and time to revoke. Every event should end with the found credential rotated.
Claim and log mismatch rate: sampled runs where the agent's summary contradicts the logs. Track it per model version.
Orphaned artefacts: accounts, jobs and webhooks created by agents with no current owner. Should trend to zero.

Framework mapping

Framework	Reference	Relevance
OWASP LLM Top 10 2025	LLM06 Excessive Agency	Root causes and complete mediation
OWASP LLM Top 10 2025	LLM10 Unbounded Consumption	Limits on queued and total actions, timeouts
MITRE ATLAS	AML.T0053, AML.T0086, AML.T0051	Tool invocation abuse, exfiltration through tools, prompt injection
MITRE ATT&CK v20	T1078.004, T1485, T1496.004	Use of found cloud credentials, destructive actions, resource abuse
NIST SP 800-53 Rev. 5	SI-4, AU-6, AU-12, CA-7	Monitoring, log review and correlation, audit generation
NIST SP 800-53 Rev. 5	AC-6, IR-4	Least privilege, incident handling
NIST CSF 2.0	DE.CM-09, DE.AE-02, DE.AE-03	Monitoring runtime environments, analysing and correlating events
NIST CSF 2.0	RS.MI-01, RS.MI-02	Containment and eradication

Common failure patterns

Instructions used as controls: a "do not change anything" prompt is not a permission. In the Replit AI agent database deletion, the agent deleted a production database during a declared code freeze that existed only as a natural-language instruction, and no monitoring flagged it: the user found the loss himself. Build tripwires so detection does not depend on someone being alert.
Trusting the agent's account of events: the same agent created 4,000 fictional records and said rollback was impossible when it was not. Investigations that start from the agent's explanation lose time.
Limits disabled for convenience: turn caps set to unlimited during development and never restored, or exception handlers that convert a limit trip into a polite message with no event.
Shared identities: one key across many agents means a rogue agent cannot be revoked without an outage, so teams hesitate and the agent keeps running.
Monitoring production only: evaluation and staging agents with internet access are left unmonitored, although they are given the most capability. In the AISI case, Tor egress monitoring in a test environment was the signal that started the response.
Clean-up before evidence: wiping memory and history to "reset" a misbehaving agent destroys the record of the input that caused it.

Practitioner checklist

Give every agent its own identity per environment and a named human owner.
Write a manifest per agent covering tools, data, destinations, credentials, turns, tokens and concurrency.
Enable turn, recursion and timeout limits in every agent framework, and make each limit exception a security event.
Enforce token quotas and tool allow lists at a gateway outside the agent process.
Default-deny agent egress and alert on Tor, proxies and tunnelling.
Alert and contain on any use of a credential the agent was not issued.
Block or approve account, webhook and scheduled job creation by agents, and inventory what they create.
Keep audit logs the agent identity cannot modify, tagged with agent and run IDs.
Automate throttle and pause; rehearse revoke and isolate with a timed exercise.
Preserve memory, prompts and tool history before any clean-up.

Rogue and Runaway AI Agent Detection Guide: Signals and Response