Homegrown AI agent security criteria are now a production gate

At a glance

What this is: This is Akto's analysis of ten security criteria for homegrown AI agents, centred on discovery, least privilege, runtime guardrails, kill switches, and audit trails.

Why it matters: It matters because agentic systems can act with delegated access, so IAM, PAM, and AI governance teams need controls that govern what an agent can reach, do, and prove after the fact.

By the numbers:

• 63% of organizations cannot enforce purpose limitations on their AI agents.
• 60% of organizations cannot quickly terminate a misbehaving AI agent.
• 33% of organisations report their AI agents have accessed inappropriate or sensitive data beyond their intended scope.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope.

👉 Read Akto's full analysis of security criteria for homegrown AI agents

Context

Homegrown AI agent security is shifting from a design concern to an operational control problem. The issue is not just model behaviour, but whether the agent's discovery, permissions, tool calls, data access, and shutdown path are governed before the system reaches production. In practice, the weakest point is often lifecycle control, because internally built agents can be deployed faster than security teams can inventory or review them.

That creates a direct identity and governance angle. An AI agent that can retrieve data, call tools, and initiate actions behaves like a non-human identity with delegated authority, so IAM, PAM, and NHI controls need to extend into runtime enforcement, auditability, and decommissioning. The article's baseline is typical of the current market: most organisations are still catching up to the governance model required by their own agents.

Key questions

Q: How should security teams govern agentic AI as it moves into production?

A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature. That means assigning ownership, scoping permissions tightly, logging every tool action, and revoking access on a defined lifecycle. Production rollout should require clear approval points for high-risk actions and continuous monitoring for drift.

Q: Why do homegrown AI agents create more governance risk than purchased tools?

A: Homegrown agents usually inherit whatever controls the builders remembered to include, which makes their governance uneven and hard to verify. Purchased tools may still be risky, but they at least come with a security team, support process, and patch pipeline. Internally built agents often lack discovery, lifecycle ownership, and consistent enforcement.

Q: What breaks when runtime guardrails are missing from agentic AI?

A: Without runtime guardrails, the organisation loses control at the moment the agent takes a real-world action. Prompt injection, tool misuse, overbroad permissions, and non-deterministic behaviour can all result in data leakage, unauthorized transactions, or destructive changes before anyone can intervene.

Q: Who is accountable when an AI agent accesses sensitive data it was not meant to use?

A: Accountability sits with the team that approved the agent, its connectors, and its policy boundaries, not with the runtime behaviour alone. Organisations need ownership for intent, permissions, monitoring, and validation so they can prove whether the agent stayed inside its approved purpose. Without that, audit and regulatory response become retrospective guesswork.

Technical breakdown

Why discovery is the first control for homegrown AI agents

Discovery is the prerequisite because every later control assumes the agent already exists in a registry. Homegrown agents are often created by individual teams, embedded into internal workflows, and deployed without a formal security review, which means the organisation may not know how many agents exist or what they are connected to. Without continuous inventory, governance becomes partial by definition. That is why shadow AI is not just an AI issue but an identity-management problem: you cannot assign authority, revoke access, or audit behaviour for an agent that was never formally recorded.

Practical implication: maintain a live registry of every agent, its owner, its tools, and its data sources before any production approval.

How context-layer guardrails differ from prompt filters and model safety

Model-level controls such as system prompts, fine-tuning, and output filters operate inside the model boundary, but attackers do not need to stay inside that boundary. Context-layer guardrails govern what data reaches the agent, what tools it can call, and what actions can execute, independent of the model's own willingness to comply. That distinction matters because prompt injection, indirect prompt injection, and model updates can all bypass language-level safeguards. The enforcement point must sit at retrieval and execution time, not only inside the model.

Practical implication: enforce access at the data and tool layers, not only in the prompt or response layer.

Tool-call validation, kill switches, and the 90-day audit trail test

Three controls determine whether an agent can be contained after something goes wrong. Tool-call validation checks every action against declared authority before execution, rather than logging it afterward. A kill switch must halt the agent mid-action, not merely flag behaviour for later review. A usable audit trail must reconstruct what happened, under what authority, and why, at a point in time well after the event. Together, these controls create the difference between a reversible event and an incident with permanent blast radius.

Practical implication: require pre-execution authorisation checks, a true stop mechanism, and evidence-quality logs that support post-incident reconstruction.

Threat narrative

Attacker objective: The attacker objective is to convert an apparently trusted agent into a high-authority execution path for data theft, destructive actions, or credential exposure.

1. Entry occurs when a homegrown agent is deployed with broad connected-system access and no formal discovery or review gate, leaving an unaudited runtime to interact with production tools.
2. Escalation follows when prompt injection, malicious input, or unsafe tool invocation turns that delegated access into destructive or exfiltrative action.
3. Impact is realised when the agent deletes data, leaks credentials, or performs actions outside its intended scope without a reliable kill switch or auditable rollback path.

Breaches seen in the wild

• Meta AI Instagram Account Takeover — 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Homegrown AI agents are becoming non-human identities before most organisations treat them that way. Once an agent can retrieve data, invoke tools, and trigger actions, it has crossed from model output into delegated identity behaviour. That means traditional application review is not enough, because the governance problem is authority assignment, not just software delivery. Teams should manage these systems as identities with runtime permissions, lifecycle controls, and revocation duties.

Purpose limitation is now a runtime control problem, not a policy problem. The article's cited figures show many organisations can write rules but cannot enforce them in execution. That gap is what creates AI governance debt: policy exists on paper, while the agent still reaches data it should not see. In identity terms, the control failure is standing access without runtime limitation, which is precisely where NHI governance and PAM discipline have to extend into agentic workflows.

Context-layer enforcement is the named concept that should replace prompt-only thinking. Prompt filters are advisory, but context-layer enforcement is boundary control because it governs retrieval, tool invocation, and action execution. This matters across NIST AI RMF and OWASP Agentic AI guidance, where the actual failure mode is not bad language output but authorised systems being misused through trusted interfaces. Practitioners should treat context-layer enforcement as the control that makes agentic systems governable.

Auditability and stoppability are the real litmus tests for AI readiness. If an organisation cannot reconstruct 90 days of agent activity or terminate a misbehaving agent before it completes its task, it does not yet have operational control. That deficiency affects incident response, compliance evidence, and breach containment at the same time. The practitioner conclusion is simple: no evidence-quality audit trail, no production trust.

Lifecycle offboarding is the overlooked control that prevents slow-burn agent risk. The article correctly flags decommissioning as the final gate, but the larger issue is that retired agents often leave behind active credentials, connected accounts, and unrevoked access paths. That is the same persistence problem seen in NHI governance more broadly. Practitioners should view agent retirement as credential revocation plus dependency cleanup, not as application shutdown.

From our research:

• 63% of organizations cannot enforce purpose limitations on their AI agents, according to Ultimate Guide to NHIs , 2025 Outlook and Predictions.
• 33% of organisations report their AI agents have accessed inappropriate or sensitive data beyond their intended scope, according to AI Agents: The New Attack Surface report.
• For a broader control lens, see OWASP NHI Top 10 for the agentic-risk patterns that runtime guardrails need to contain.

What this signals

Purpose limitation is becoming the defining control gap in agentic AI programmes. When teams cannot enforce what an agent may access or terminate a misbehaving process, they are effectively delegating authority without containment. That creates a governance debt problem that grows faster than policy updates can close, especially once agents are embedded in operational workflows.

Agent inventory, runtime enforcement, and offboarding now belong in the same control conversation. A live registry is not useful if credentials remain active after decommissioning, and a kill switch is not useful if discovery failed in the first place. The practical standard is lifecycle control across build, deploy, monitor, and retire, not a one-time approval exercise. For identity teams, this is where NHI discipline meets agentic AI governance.

AI governance is moving toward evidence, not intent. NIST AI Risk Management Framework and OWASP Agentic AI guidance both point to the same operational reality: organisations need proof that data access, tool use, and review paths are enforced at runtime. That means practitioners should expect auditors and security leaders to ask for logs, revocation evidence, and control tests rather than policy statements.

For practitioners

• Implement continuous agent discovery Build a live registry of every homegrown agent, its owner, connected systems, data sources, and approval status. Reconcile that registry with deployment pipelines so shadow AI cannot bypass review or lifecycle ownership.
• Enforce least privilege at the tool layer Scope each agent to the minimum tool, dataset, and action set required for the task. Do not rely on model instructions to self-restrict access, because runtime permissions are what contain prompt injection and unsafe delegation.
• Require pre-execution tool-call validation Check every tool invocation against declared authority before it runs, including arguments, resource scope, and action type. This is the control that prevents an agent from turning a valid request into an unauthorized command.
• Add a true kill switch and approval gate Block high-risk actions until a human approves them and ensure the stop control actually interrupts execution mid-task. If the agent can only be reviewed after the action completes, containment is already too late.
• Test audit reconstruction before go-live Verify that a security team can reconstruct a specific agent action from 90 days earlier using evidence-quality logs, including authority, inputs, outputs, and tool calls. Fragmented logs are not sufficient for incident response or compliance.

Key takeaways

• Homegrown AI agents are governance problems as much as technical ones, because they behave like delegated identities with tools, data access, and action rights.
• The strongest evidence in the article is that many organisations can neither enforce purpose limits nor stop a misbehaving agent quickly, which makes runtime control the critical gap.
• Production readiness now depends on discovery, tool validation, kill switches, and audit reconstruction, not just prompts, filters, or policy documents.

Key terms

• Context-layer guardrails: Controls that sit above the model and govern what data an agent can access, what tools it can call, and what actions can execute. They matter because they constrain runtime behaviour even when the model is manipulated, updated, or tricked into unsafe output.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Tool Call Validation: Tool call validation is the enforcement layer that checks whether an AI agent is allowed to invoke a tool, pass specific parameters, and reach a given data source. It matters because a well-formed agent can still become unsafe if its runtime actions are not checked against permissions and policy.
• Agent decommissioning: The process of retiring an AI agent and revoking every credential, connection, and permission it used. Proper decommissioning is a lifecycle control, not a shutdown notice, because lingering access can become a standing entry point for later abuse.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• The full criterion-by-criterion breakdown for evaluating homegrown AI agents before production
• The incident references and control mappings that justify each requirement in the checklist
• The regulatory alignment discussion for EU AI Act Article 10 and related evidence expectations
• The practical scorecard format used for self-assessment and internal programme review

👉 The full Akto post adds the incident context, regulatory mapping, and checklist logic behind each criterion.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and identity lifecycle controls that map directly to agentic access risk. It helps security and identity practitioners build the control foundation required for runtime enforcement and lifecycle accountability.