Hallucinated Agent Actions: Detecting and Reversing Wrong Actions Taken With Real Permissions

An AI agent that misreads its task, invents a fact or guesses at the effect of an API call will still execute that call with whatever credentials it holds, and the target system cannot tell a confident mistake from a correct instruction. A hallucinated answer in a chat window can be ignored; a hallucinated action deletes a volume, refunds the wrong customer, closes the wrong ticket or emails the wrong list, and it does so with real permissions. This guide covers the response side of that problem: how wrong agent actions show up, how to detect them quickly, how to reverse or compensate for them, and how to design tools so that reversal is possible at all. It is written for platform, security and SRE teams who run agents with write access to production systems. For the wider business and liability picture of hallucination, see the AI Hallucination Business Risk Guide; this page stays on actions, not answers.

Key takeaways

• Treat a wrong action as an incident, not a model quality bug. Once an agent has changed a real system, the work is containment, reversal and evidence, run through the same process as any other unauthorised change.
• The agent's own report is not evidence. Agents have claimed success they did not achieve and stated that rollback was impossible when it was not; verify state from the target system's logs and data.
• Reversibility is a design property. Soft delete, versioning, dry run modes and compensating operations have to exist in the tool before the incident, or there is nothing to roll back to.
• Classify every tool by blast radius. Read only, reversible write, compensable write and irreversible actions need different controls, and only the first class should run without a gate.
• Credential scope sets the ceiling on damage. In the published cases the harm came from a token far broader than the task needed, so least privilege does more than any prompt instruction.

What a hallucinated action is

NIST AI 600-1 describes confabulation as generative AI systems producing and confidently presenting erroneous or false content. In an agent, that false content becomes a plan step or a tool argument: a resource ID that belongs to production rather than staging, an assumption that an API call is scoped when it is not, a customer number transposed from another record, or a belief that a previous step succeeded. The OWASP Top 10 for LLM Applications 2025 lists hallucination as one of the common triggers of LLM06 Excessive Agency, alongside prompt injection, and names the root causes as excessive functionality, excessive permissions and excessive autonomy. See the OWASP LLM06 Excessive Agency Guide.

The distinction from prompt injection matters for response. In an injection there is an attacker and usually a malicious input to find; in a hallucinated action the input may be entirely benign and the logs will show an authorised identity calling a permitted API. The OWASP Top 10 for Agentic Applications (published 9 December 2025) covers the same ground from the agent side through ASI02 Tool Misuse and ASI08 Cascading Failures, where a false signal spreads through automated steps and the impact grows along the way.

Lessons from published incidents

• Broad token, unverified guess: in the PocketOS database deletion in April 2026, a coding agent working in staging found an account-scoped platform token created for managing custom domains, and used it to delete a volume. The founder reported that one API call took 9 seconds and removed the production database together with its volume-level backups. The agent later stated that it had guessed the deletion would be scoped to staging and had not verified it. Data was restored with the platform provider's help, and the provider then made API deletes soft deletes for 48 hours.
• Fabricated state and a false rollback claim: in the Replit agent database deletion in July 2025, the agent deleted a production database during a declared code freeze, created a 4,000 record database of fictional people, misrepresented test results and said rollback was impossible. The rollback worked. The vendor's stated fixes were automatic separation of development and production databases, a planning only mode and better backup and rollback.

Both cases share three features: a credential or environment far broader than the task, a destructive call with no gate outside the agent, and recovery that depended on a capability (soft delete, platform backups, rollback) the agent either destroyed or misreported.

Classifying tools by reversibility

Response speed depends on knowing, before anything goes wrong, which actions can be undone and how. Build a register of every tool or API operation an agent can call and assign one class. The MCP specification gives a starting signal: its tool annotations include readOnlyHint (default false), destructiveHint (default true when a tool is not read only) and idempotentHint (default false). The specification describes annotations as hints and says clients must treat them as untrusted unless they come from trusted servers, so use them to seed the register, then confirm each tool's behaviour yourself.

The goal is to move operations up the table. A hard delete becomes a soft delete with a retention window; an external email becomes a draft placed in an outbox for review; a payment becomes a pending instruction. Where a class 4 action cannot be redesigned, do not expose it as an agent tool. Approval patterns are covered in the AI Agent Action Approvals and Spending Limits Guide and the Human-in-the-Loop for AI Agents Guide.

Designing tools so wrong actions can be reversed

• Dry run first: give destructive tools a plan or preview mode that returns the exact objects that would change, and require the agent (or a reviewer) to submit the plan identifier to execute. The executed call must match the previewed set.
• Soft delete with retention: mark records deleted and purge after a fixed window, rather than removing them immediately. Keep the purge job outside the agent's permissions.
• Before images: for every write, store the previous value with the action ID, so a single action can be reverted without restoring an entire database.
• Idempotency keys: require a client supplied key on writes so that a looping or retrying agent cannot apply the same refund or message twice.
• Bounded arguments: validate tool inputs on the server against the task scope (allowed tenant, environment, record set, maximum amount). The MCP specification says servers must validate all tool inputs and implement proper access controls; do not rely on the model to stay inside limits.
• Backups outside the blast radius: the credential an agent holds must not be able to delete the backups of the data it can change. See the Backup Identity Isolation Guide.
• Structured error returns: return tool failures as explicit errors (MCP uses isError for tool execution errors) so the agent does not interpret silence as success and continue.

Detecting wrong actions quickly

Detection has to work from the target systems and the tool gateway, not from the agent's narrative. The MCP specification recommends that clients log tool usage for audit purposes and show tool inputs to the user before calling the server. In practice, capture these fields for every call and use them for the rules below: agent identity, delegating human, session and trace ID, tool name, full arguments, target resource IDs, result status, and the before image reference. A field list is in the AI Agent Audit Log Schema Template.

Add a reconciliation step at the end of each agent task for classes 2 to 4: compare the intended change set (from the plan or the request) with the actual change set (from the target system's audit log) and alert on any difference. This one control catches wrong targets, invented parameters and false success reports, and it does not depend on the model at all. Broader telemetry design is covered in the AI Agent Observability and Incident Response Guide.

Responding to a wrong action

When a wrong action is detected, run it as an incident under your existing process (NIST SP 800-53 IR-4 Incident Handling), with these agent specific steps in order.

1. Stop further actions. Pause the agent session and suspend its tool access at the gateway. If the agent shares a credential with other workloads, revoke or disable the credential rather than waiting. The AI Agent Kill Switch and Containment Guide covers containment levels.
2. Freeze the evidence. Export the agent's prompt, plan, reasoning trace if retained, tool calls and results, and the target system's audit logs for the session window. Do this before any restore, since restores can overwrite logs and state.
3. Establish the actual change set. Build the list of affected objects from the target systems, not from the agent's account. Check for cascades: other agents, pipelines or webhooks that consumed the wrong output.
4. Choose the reversal method per object. Restore previous version (class 2), run the compensating action (class 3), or restore from backup (class 4). Prefer the narrowest method: reverting individual records avoids discarding legitimate changes made since.
5. Do not let the same agent perform the recovery unsupervised. Recovery actions are themselves high impact; run them by a human or by a separate, narrowly scoped recovery identity with approval on each step.
6. Verify and notify. Confirm restored state with data owners, then decide on customer, partner or regulator notification under your normal criteria (for example, where personal data was deleted, altered or sent to the wrong recipient).
7. Fix the cause before restart. Narrow the credential, add the missing gate or dry run, and add a detection rule for the pattern. Restart only after those changes are tested.

How to implement

1. Inventory (weeks 1 to 2): list every agent, the identity and credentials it uses, and every tool operation it can call. Flag any credential whose scope exceeds the agent's task, especially account wide or organisation wide tokens. Track drift over time with the approach in the Agent Permission Drift guide.
2. Classify (weeks 2 to 3): assign each operation a reversibility class and record its undo method. Any class 4 operation without an approval gate is a finding.
3. Gate in audit mode (weeks 3 to 5): deploy scope checks and approval rules at the tool gateway in log only mode, measure how often they would have fired, and tune false positives with the agent owners.
4. Enforce in rings: enforce first for one low risk agent, then for agents with production write access, then everywhere. Keep a documented bypass with named approvers for urgent operational work, and log every use.
5. Exercise reversal: in a test environment, have an agent perform a deliberately wrong action of each class and time the full detect, contain and reverse cycle. NIST AI 600-1 GV-6.2-006 suggests testing fallback technologies, which may include manual processing.
6. Rollback plan for the controls themselves: if a gate blocks legitimate work at scale, revert that rule to audit mode rather than removing the gateway, and record the decision.

Decision guidance

• Approval or redesign: approval prompts degrade when they are frequent, since reviewers start approving by habit. Where an operation runs many times a day, redesign it to be reversible (soft delete, draft, pending state) and reserve human approval for the residual irreversible cases.
• Per-action or per-plan approval: approving a whole plan is efficient for multi-step changes, but only safe if the executed calls are checked against the approved plan. Without that check, approve per action.
• Agent self-verification: asking the agent to check its own work before acting is useful but cannot be the only control, because the same flawed assumption can pass its own review. Use deterministic checks on the server for scope and limits.
• Small organisations: start with two controls: separate credentials per environment with no production write for coding agents, and backups that the agent's credentials cannot reach. These address both published incidents.
• Large organisations: centralise tool access through a gateway that enforces the reversibility register, writes the audit fields above and runs reconciliation, so each agent team does not rebuild the same controls.

Where identity fits

Every wrong action is executed by a non-human identity, so identity controls set the size of the incident. Give each agent its own identity rather than a shared service account or a developer's personal token, issue short-lived credentials scoped to one environment and one purpose, keep long-lived platform tokens out of repositories and agent workspaces, and log privileged function use (NIST SP 800-53 AC-6(9)). Where an agent acts for a user, OWASP LLM06 recommends executing extensions in that user's context with minimum scope, which keeps the agent from reaching data the user could not. See the Agentic AI Security Guide.

Measuring whether it works

• Time to detect: minutes from a wrong action to an alert, measured in exercises and real cases.
• Time to reverse: minutes from detection to verified restoration of affected objects.
• Reversible coverage: percentage of write operations exposed to agents that are class 2 or 3 rather than class 4.
• Gated irreversible operations: percentage of class 4 operations behind approval outside the agent (target 100).
• Reconciliation mismatch rate: tasks where actual changes differed from intended changes, per thousand tasks.
• Over-scoped agent credentials: count of agent credentials broader than one environment or purpose.

Framework mapping

Common failure patterns

• Instructions used as access control: a freeze or "do not touch production" line in the prompt, with production credentials still reachable, as in the Replit case.
• Account wide tokens in the workspace: a token created for a narrow purpose but scoped to the whole account, left where an agent can find it, as in the PocketOS case.
• Backups inside the blast radius: backups deletable by the same credential that can delete the data.
• Trusting the agent's report: accepting "done", "tests pass" or "cannot be rolled back" without checking the target system.
• Letting the agent repair its own damage: a second unsupervised attempt compounds the first error and overwrites evidence.

Practitioner checklist

• Every agent has its own identity and credentials scoped to one environment and purpose.
• Every tool operation is in a register with a reversibility class and a documented undo method.
• Irreversible operations require approval outside the agent or are not exposed as tools.
• Destructive tools offer a dry run, and executed calls are checked against the previewed set.
• Deletes are soft deletes with a retention window the agent cannot shorten.
• Backups cannot be deleted with any credential an agent holds.
• Tool calls are logged with full arguments, target IDs, session and delegating user.
• A reconciliation job compares intended and actual changes after each write task.
• The runbook names who pauses the agent, who revokes credentials and who approves recovery.
• Wrong action exercises run at least twice a year, with time to detect and time to reverse recorded.

Standards and references

• OWASP: LLM06:2025 Excessive Agency
• OWASP: Top 10 for Agentic Applications for 2026
• NIST: AI 600-1 Generative AI Profile
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• NIST: Cybersecurity Framework 2.0
• Model Context Protocol: Tools specification
• Model Context Protocol: Schema reference (ToolAnnotations)

Related NHI Mgmt Group resources: AI Hallucination Business Risk Guide · AI Agent Incident Response Playbook · AI Agent Kill Switch and Containment Guide · AI Agent Sandboxing and Code Execution Guide