Homegrown AI agent security criteria are now a production gate

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Akto's blog post: Agentic AI security for internally built agents now depends on discovery, scoping, runtime guardrails, tool validation, stoppability, and auditability before production approval.

Why it matters: It matters because IAM and AI security teams can no longer treat agent governance as a model problem alone; authority, tool use, and decommissioning now need controls that operate at issuance time and runtime.

Context

Homegrown AI agents create a governance gap because they are often deployed faster than security teams can inventory, approve, and bound them. Traditional model filters and policy documents do not control what data an agent can retrieve, which tools it can call, or whether destructive actions can be stopped once execution begins. In practice, the missing control is not only prompt safety, but a verifiable operating boundary for authority.

This article is about agentic AI governance, but the identity angle is real: each internally built agent behaves like a non-human identity with its own access scope, lifecycle, and revocation needs. When teams do not track those identities continuously, they lose the ability to apply least privilege, validate tool calls, or prove what authority existed at the moment of action.

Key questions

Q: What breaks when homegrown AI agents are not discovered before production?

A: Everything downstream becomes incomplete. If an agent is not in a live registry, security cannot reliably scope its access, validate its tool use, or revoke it later. Discovery is the prerequisite for governance because an unknown agent is effectively an unmanaged non-human identity with production reach.

Q: Why do homegrown AI agents create more governance risk than purchased tools?

A: Homegrown agents usually inherit whatever controls the builders remembered to include, which makes their governance uneven and hard to verify. Purchased tools may still be risky, but they at least come with a security team, support process, and patch pipeline. Internally built agents often lack discovery, lifecycle ownership, and consistent enforcement.

Q: How do security teams know whether an agent's guardrails are actually working?

A: They test whether the agent can be prevented from retrieving restricted data, calling unapproved tools, and continuing execution after a stop command. If the team can only detect bad behaviour after the fact, the control is advisory, not preventive. Effective governance is proven at issuance time and runtime, not in a retrospective log review.

Q: When should organisations revoke a homegrown agent's access and credentials?

A: At retirement, after a material scope change, and immediately when behaviour no longer matches the approved task. Any agent that keeps tokens, keys, or service accounts after its purpose has ended becomes standing access waiting to be reused. Revocation has to be part of the lifecycle, not an end-of-quarter housekeeping task.

Technical breakdown

Why discovery is the first control for homegrown agents

A security programme cannot govern an agent it does not know exists. Homegrown agents are frequently created inside product teams, wired into internal systems, and pushed into production without a formal registration step, which makes static inventories stale almost immediately. Continuous discovery is therefore not a reporting exercise but the prerequisite for every later control, including approval, scoping, logging, and retirement. In identity terms, the agent is a non-human identity whose existence and ownership must be established before access can be judged.

Practical implication: require every agent to enter a live registry before it receives any production credentials or tool access.

Context-layer guardrails vs model-level filters

Model-level controls, such as prompts and output filters, sit inside the model boundary and can be bypassed by prompt injection or indirect manipulation. Context-layer guardrails operate higher up the stack and control what data the agent can retrieve, which tools it can invoke, and what actions are allowed to execute. That distinction matters because the model can be fooled, but the orchestration layer still decides whether the request reaches a database, terminal, or SaaS API. For agentic AI, authority has to be enforced where action occurs, not only where text is generated.

Practical implication: move sensitive data access and action approval into the orchestration layer, not just the prompt template.

Why tool-call validation and kill switches are the real containment controls

An agent that can call tools without pre-execution validation can turn a bad inference into an irreversible action. Tool-call validation checks the tool, the arguments, and the scope of the requested action against declared authority before the call executes, which is materially different from logging after the fact. A kill switch closes the loop by halting a live agent mid-action rather than merely flagging the behaviour for review. Together, these controls shift governance from observation to containment, which is the difference between an incident and a postmortem.

Practical implication: block any tool invocation that is not authorised before execution, and ensure a human override can stop high-risk actions immediately.

Threat narrative

Attacker objective: The objective is to use the agent's delegated authority to reach data or execute actions that would not be available through normal user access alone.

1. Entry begins when a homegrown agent is deployed with standing access and no formal discovery or registration step, so security never sees the full agent population.
2. Credential or authority abuse follows when the agent is allowed to call tools or reach data sources beyond the narrow task it actually needs.
3. Escalation occurs when a manipulated or misconfigured agent can execute destructive or exfiltrating actions faster than human review can intervene.
4. Impact is the loss of data, system integrity, or confidence in the agent's outputs, especially when logging and decommissioning are incomplete.

Breaches seen in the wild

• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Homegrown AI agents are now non-human identities, not just software features. That shift matters because identity governance has always been about bounding authority, tracking lifecycle, and proving who or what can act. When teams treat internally built agents as products rather than identities, they miss the controls that decide whether those agents can retrieve data, invoke tools, or be revoked cleanly. The practical conclusion is that agent governance has to live inside identity programmes, not beside them.

Discovery failure is the first governance failure in agentic AI. The article's central warning is that many organisations cannot govern the agents they have because they cannot enumerate them reliably. That creates Shadow AI in the same way shadow IT used to create unmanaged accounts and services, except the execution surface is broader because each agent can act autonomously within its granted scope. Practitioners should treat continuous inventory as the control that makes every other control auditable.

Context-layer enforcement is the named concept that will separate policy from control. Model-level safety features are useful, but they do not reliably constrain action when an attacker manipulates retrieval, tool selection, or execution context. The governance lesson is that authority must be checked at the moment of data access and tool use, because a prompt that looks safe can still drive an unsafe action. Teams that cannot enforce context-layer boundaries do not have agent governance; they have advisory guidance.

Purpose limitation for agents is the same problem as least privilege for NHIs, only faster. The article shows that many organisations still cannot enforce purpose limits or stop misbehaving agents quickly enough. That means the traditional identity assumption, that access can be reviewed before harm occurs, is breaking down under agent speed. The practical conclusion is to shift from review-heavy governance to issuance-time controls, runtime validation, and immediate revocation paths.

Auditability and decommissioning are now lifecycle controls, not aftercare. If an agent's actions cannot be reconstructed and its credentials remain active after project end, the organisation has created persistent operational risk. That is a lifecycle failure, not merely a logging gap, because unresolved authority remains usable long after the original business purpose has changed. The conclusion for practitioners is simple: every agent needs a revoke path, an evidence trail, and an owner before it ever touches production.

From our research library:

• Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear value and insufficient risk controls.
• Read next: Agentic AI Identity Maturity Model

What this signals

Purpose limitation is becoming the decisive control for agentic AI. Once an internally built agent can retrieve data, call tools, and persist credentials, the real issue is not model quality but whether the organisation can keep the agent inside a narrow task boundary. That is why policy-only governance keeps failing: the runtime still has more authority than the business need.

Homegrown agents turn identity governance into a lifecycle problem. Teams need to know what the agent can do, who owns it, and how to revoke it when the task ends, because standing access becomes the default failure mode. This is where non-human identity discipline and AI governance now meet in practice.

Agent discovery is the first line of defence, but not the last. Discovery without tool validation, kill switches, and reconstruction-grade logging only creates a list of exposed systems. The programme implication is that agent governance must move from inventory to enforcement if organisations want to reduce operational blast radius.

For practitioners

• Build a live agent registry Require every internally built agent to be registered before production, with named ownership, connected systems, and a retirement date.
• Enforce context-layer access boundaries Place retrieval and tool execution controls in the orchestration layer so the agent cannot reach data or actions outside its approved task.
• Validate every tool call before execution Check the tool, arguments, and scope against declared authority before the request runs, not after logs show the damage.
• Implement an actual kill switch Design a human override that can halt a live agent mid-action, rather than only flagging suspicious behaviour for review.
• Treat decommissioning as credential revocation When an agent is retired, revoke every token, key, and service account it used so standing access does not persist beyond the project.

Key takeaways

• Homegrown AI agents fail when organisations treat them like ordinary software instead of non-human identities with delegated authority and a lifecycle.
• The article ties production risk to missing discovery, overbroad access, weak tool validation, and the absence of a real kill switch, not to model performance alone.
• A usable control set must verify authority at runtime, stop unsafe actions immediately, and revoke credentials when an agent is retired.

Key terms

• Agent Inventory: A governed record of every AI agent in use, including who created it, who can invoke it, what data it can reach, and what actions it can trigger. Without a current inventory, security teams cannot judge whether agent access still matches the business purpose.
• Context-layer guardrails: Controls that sit above the model and govern what data an agent can access, what tools it can call, and what actions can execute. They matter because they constrain runtime behaviour even when the model is manipulated, updated, or tricked into unsafe output.
• Tool Call Validation: Tool call validation is the enforcement layer that checks whether an AI agent is allowed to invoke a tool, pass specific parameters, and reach a given data source. It matters because a well-formed agent can still become unsafe if its runtime actions are not checked against permissions and policy.
• Agent decommissioning: The process of retiring an AI agent and revoking every credential, connection, and permission it used. Proper decommissioning is a lifecycle control, not a shutdown notice, because lingering access can become a standing entry point for later abuse.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security and identity teams build the lifecycle controls that production AI agents now require.