Continuous security testing is essential for AI agent governance

At a glance

What this is: This is an analysis of why AI agent security testing must be continuous, with the key finding that snapshot assessments miss prompt, model, tool, and runtime drift.

Why it matters: It matters because IAM, IGA, and security teams need controls that keep pace with changing agent privileges, tool access, and behavioural scope across autonomous and non-autonomous systems.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.

👉 Read Akto's analysis of continuous security testing for AI agents

Context

AI agent security testing is the practice of validating whether an agent still behaves safely after its prompt, model, tools, memory, or permissions change. The problem is not that testing is unnecessary, but that the security posture of an agent can shift between one review cycle and the next, especially when changes happen outside normal deployment gates.

For IAM and security teams, this creates a governance gap that looks familiar from NHI programmes but behaves differently at runtime. The source article argues that continuous validation, not one-time approval, is required because agent behaviour and exposure can drift daily or even hourly, which makes static assurance increasingly stale before it is acted on.

Key questions

Q: How should security teams test AI agents after prompts, models, or tools change?

A: They should test AI agents every time a meaningful change occurs, not on a calendar alone. Prompt edits, model swaps, new tools, memory sources, and permission changes can all alter behaviour enough to invalidate an earlier result. The safest model is continuous validation tied to the live version of the agent, plus runtime monitoring after release.

Q: Why do point-in-time assessments fail for AI agent governance?

A: They fail because the assessment is usually completed against a version that has already changed. Agent behaviour can shift without a formal code release, so the old report may no longer describe the current prompt, model, tool set, or permissions. That creates false assurance and leaves teams unaware of drift that has already happened.

Q: What breaks when AI agents are given broad inherited permissions?

A: Broad inherited permissions break the assumption that access is tied to a narrow business need. The result is larger blast radius, weaker accountability, and faster propagation of mistakes or abuse across connected systems. A single compromised or misconfigured agent can then touch far more data and workflows than the original task required.

Q: What should organisations do when AI agent security is changing faster than review cycles?

A: They should shift from periodic approval to continuous governance. That means automated pre-deployment red teaming, runtime guardrails, change-triggered retesting, and access recertification for tools and data sources. The goal is to govern the agent as a live identity with evolving scope, not as a one-time software release.

Technical breakdown

Why point-in-time testing fails for AI agents

Point-in-time testing assumes the system under test is stable enough that a security verdict remains meaningful after the report is issued. AI agents break that assumption because prompt edits, model swaps, new tool integrations, memory updates, and permission changes can alter behaviour without a conventional code release. In practice, the security boundary is moving while the assessment is being written. That means a clean test result can describe a version that no longer exists in production, which creates a false sense of assurance for security and compliance teams.

Practical implication: Treat agent security as a changing control state, not a single certification event.

How runtime risks appear in tool-using and autonomous agents

Runtime risk emerges when an agent acts in a live environment where inputs, retrieved context, and tool availability are unpredictable. A lab test cannot reproduce every customer conversation, poisoned memory source, or multi-agent workflow. The article correctly separates non-deterministic behaviour from deterministic application logic: once an agent can chain decisions and tool calls, the attack surface includes prompt injection, tool misuse, goal hijacking, and context poisoning that may only appear during actual execution. That is why runtime guardrails and monitoring are complementary to pre-deployment red teaming rather than replacements for it.

Practical implication: Monitor live agent actions and tool calls, not just pre-release test results.

Why MCP and permission drift require continuous validation

New tool integrations and MCP servers expand what an agent can do, but they also expand trust boundaries, data exposure paths, and lateral movement options. The article highlights a subtle but important point: permissions tend to accumulate while rarely being removed after the original use case passes. That creates permission drift, where a once-narrow agent becomes materially more privileged over time. In agentic environments, continuous testing is therefore as much about access governance as it is about security testing, because the effective identity posture changes whenever tools or privileges change.

Practical implication: Re-test every new tool, integration, and privilege change before the agent reaches production scope.

Threat narrative

Attacker objective: The objective is to turn a trusted agent into a live execution path for unauthorised access, data exposure, or destructive action.

1. Entry occurs when attackers or users exploit prompt injection, poisoned context, or a newly added tool path inside a live agent workflow.
2. Escalation happens when the agent inherits broader permissions through tool chaining, permission drift, or an MCP integration that widens access beyond the original validation scope.
3. Impact follows when the agent performs unauthorised actions, exposes sensitive data, or triggers downstream systems based on stale guardrails or outdated assessments.

Breaches seen in the wild

• Meta AI Instagram Account Takeover — 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Static security assurance is structurally mismatched to agentic systems. A point-in-time assessment assumes the object being tested remains close enough to the tested version that the result still means something later. That assumption fails when prompts, models, tools, and permissions change continuously. The implication is not just that testing should happen more often, but that assurance has to track the live identity state of the agent, not a historical snapshot.

Continuous validation is the governance layer agentic systems actually require. The article correctly shows that discovery, red teaming, and runtime protection only work when they are linked into a loop. This aligns with OWASP Agentic AI risks and NIST AI RMF measurement and manage functions, because the control problem is behavioural drift, not a one-time configuration flaw. Practitioners should treat continuous testing as an operating model, not a project milestone.

Permission drift is the NHI issue hidden inside agent security. AI agents often start narrow and become more capable through new tools, broader data access, and expanded delegation. That makes them look like software while behaving like evolving non-human identities. The named concept here is runtime governance gap: the difference between what the agent was approved to do and what it can do after successive changes. Teams should govern the current authority set, not the original design.

Autonomous agents break the assumption that access remains stable long enough to review. Access review was designed for conditions where privilege persists across a meaningful review period. That assumption fails when an autonomous actor can chain decisions and tools within a single runtime session, creating a moving target before any reviewer can observe it. The implication is that identity governance must distinguish between reviewable standing access and behaviour that never stabilises into a durable entitlement.

MCP expands the coordination layer and the blast radius at the same time. Once agents discover and invoke external tools through a standard protocol, a misconfigured or malicious server can affect more than one agent or workflow. That changes the identity problem from isolated tool control to shared trust infrastructure. Practitioners should view MCP as a governance boundary that needs continuous validation, not a fixed integration point.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• A second finding from the same research shows that only 44% have implemented policies to govern AI agents, which leaves most deployments with weak behavioural control even before runtime drift is considered.
• For a broader view of how agent behaviour changes the identity model, see OWASP NHI Top 10 for the risks that continuous validation needs to cover.

What this signals

Runtime governance gap: the real issue is not whether an AI agent passed a review last quarter, but whether its current prompt, model, tools, and permissions still match the approved control state. Governance programmes need change-triggered validation and live enforcement because review cadence alone cannot keep up with agent drift.

With 52% of companies able to track and audit the data their AI agents access, the remaining blind spot is large enough to distort both compliance reporting and incident response. That gap becomes more severe when new tools and MCP-style integrations expand what the agent can see and do between reviews.

Teams that already map identity controls to OWASP Agentic AI and the NIST AI Risk Management Framework will be better positioned to operationalise continuous testing, because the standard itself is only useful when measurement follows the current system state. The practical next step is to tie discovery, red teaming, and runtime guardrails into one lifecycle.

For practitioners

• Rebuild testing around every meaningful change Trigger security validation whenever prompts, models, tools, memory sources, or permissions change, rather than waiting for quarterly or annual review cycles.
• Separate pre-deployment testing from runtime enforcement Use automated red teaming to catch regressions before release, then enforce live AI guardrails that can block unsafe tool calls or sensitive data access in production.
• Version-control agent prompts and tool definitions Treat prompts, tool schemas, and agent configurations as governed artifacts that move through the same review discipline as code and infrastructure changes.
• Map agent access drift to identity governance controls Review which tools, data sources, and impersonation paths have accumulated over time, then recertify them against current business need and current risk.
• Instrument MCP integrations as trust boundaries Record which agents can discover and invoke each MCP server, then retest those relationships whenever a new server is attached or an existing one changes scope.

Key takeaways

• AI agent security cannot be treated as a snapshot exercise because prompts, tools, models, and permissions change faster than review cycles.
• The article’s core evidence is that live agent behaviour drifts into unsafe scope, which makes static assurance and stale reports a governance liability.
• Continuous testing, runtime guardrails, and access recertification are the controls that align security oversight with how agentic systems actually evolve.

Key terms

• Continuous Security Testing: A security model that revalidates an AI agent whenever its prompt, model, tools, memory, or permissions change. For agentic systems, this is not a pipeline stage but a living control that tracks behaviour as the system evolves in production.
• Runtime Guardrail: A control applied while an AI agent is operating, not just during configuration or review. Guardrails can block dangerous tool calls, require approval for sensitive actions, or stop data leakage before it reaches systems or users.
• Permission Drift: Permission drift is the gradual expansion of access beyond what was originally intended. It happens when roles, tokens, and service accounts accumulate unused rights over time, making cloud identities harder to review and more dangerous to compromise.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.

What's in the full article

Akto's full blog post covers the operational detail this post intentionally leaves for the source:

• A continuous testing lifecycle for AI agents across development, pre-deployment, and runtime enforcement
• Examples of prompt injection, goal hijacking, tool abuse, context poisoning, and permission drift scenarios
• A practical mapping of continuous testing to NIST AI RMF functions and OWASP-style agentic risk coverage
• How discovery, red teaming, and guardrails connect into a closed-loop operating model

👉 Akto's full post covers the runtime guardrail model, agent drift scenarios, and continuous validation workflow.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.