AI agent security needs safety inputs to become secure by design

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Zenity's blog post: AI safety and AI security need to converge because agent autonomy changes both the control questions and the failure modes.

Why it matters: For IAM and identity teams, this matters because AI agents behave as governable identities with runtime decisions, meaning security programmes must cover both what they do and why they do it.

Context

AI agent security is the governance problem that appears when software can choose actions, tools, and timing during runtime rather than following a fixed script. In that environment, the usual split between building a trustworthy system and securing its operations starts to break down, because the same actor must be understood through both safety and security lenses.

This article uses a New York AI Safety Meetup as the trigger for a broader point: AI agents cannot be governed well if teams only inspect their outputs or only assess model behaviour. Identity, authorisation, and decision-making are now coupled, so practitioners need a programme that can explain both the action path and the intent behind it.

That is a different operating model from traditional software security, where shift-left efforts often failed because product and security teams worked in separate lanes. The article's starting position is atypical in one respect: it argues that AI gives the community a chance to build secure-by-design discipline before the old divide hardens again.

Key questions

Q: How should security teams govern AI agents that can change actions at runtime?

A: Security teams should govern runtime AI by correlating identity, data, and intent before trusting an action path. If the system can select tools or alter its sequence mid-session, a static access policy is not enough. The control objective becomes contextual verification of what the agent is doing, why it is doing it, and whether the data touched matches the approved purpose.

Q: Why do safety controls and security controls both matter for AI agents?

A: Safety controls address whether the agent behaves in ways that remain trustworthy and aligned, while security controls address whether the agent can be attacked, misused, or pushed off course. If one side is missing, an organisation can end up with a well-aligned system that is still exploitable, or a hardened system that still behaves in unsafe ways.

Q: What are the signs that AI governance is failing in the enterprise?

A: Common warning signs include rapid growth in AI use without matching policy coverage, sensitive files being copied into personal accounts, and a large share of AI apps carrying high or critical risk. Another indicator is weak visibility into who is using which tools and what data they are sending. If teams cannot answer those questions, governance is not working as intended.

Q: What should organisations do when AI agent autonomy and secure by design conflict?

A: They should treat autonomy as a governance constraint, not just a product feature. The practical test is whether the organisation can explain, monitor, and challenge the agent's decisions without relying on assumptions that only work for fixed workflows. If it cannot, the programme needs a stronger control model before wider deployment.

Technical breakdown

Why AI agent security needs both safety and security controls

AI agents are not just models with prompts. They are systems that can act, select tools, and decide timing inside an environment, which means their risk surface includes both harmful action and harmful intent. Safety work asks whether the agent behaves in a way that stays aligned with expected outcomes. Security work asks whether the agent can be attacked, misused, or pushed into taking actions that were never intended. When those two views are split, teams miss the link between behaviour, control, and accountability.

Practical implication: assess agents as runtime actors, not as isolated models or simple automation.

What changes when autonomous decisions enter the identity model

Autonomy changes the identity problem because the actor is no longer only responding to instructions. A human, service account, or workflow can be governed through fixed rules, but an autonomous agent may choose among actions and tools based on context. That makes authorisation more dynamic and makes post-event review less reliable if teams only log outputs. The governance question becomes whether the system's decision path was acceptable, not just whether the final action was allowed.

Practical implication: extend governance from action review to decision-path review for agentic systems.

Secure by design for AI depends on shared evaluation language

The meetup highlights a basic structural issue: safety communities often evaluate model behaviour and trustworthiness, while security teams evaluate exposure, misuse, and response. Those vocabularies are different, but the same agent has to satisfy both. Without a shared evaluation language, organisations end up with aligned models that are operationally unsafe or hardened systems that still behave unpredictably. The useful middle ground is to connect evaluation, monitoring, and response around the same agent identity.

Practical implication: create one control model that joins evaluation, monitoring, and incident response for agents.

Breaches seen in the wild

• Microsoft Azure OpenAI abuse by Storm-2139: Storm-2139 used API keys leaked by Microsoft customers to hijack Azure OpenAI, bypass safety guardrails and resell access to generate harmful content.
• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI agent governance fails when safety and security are treated as separate disciplines. The article's central point is that neither alignment work nor operational security is sufficient on its own once an agent can act in the environment. Safety explains whether the behaviour is acceptable, while security explains whether the behaviour can be trusted under pressure. Practitioners need to stop treating those as parallel programmes and start treating them as one control surface.

Runtime decision-making changes the identity problem from static permissioning to behaviour assurance. An autonomous agent can select actions and tools in context, which means authorisation is no longer only a provisioning question. That makes the security question less about whether access was granted and more about whether the agent's decision path stayed within acceptable bounds. The implication is that AI governance must account for runtime discretion as a first-class control variable.

Secure by design for AI will depend on a named concept we can actually govern: decision-path assurance. That means teams need to know not only what an agent did, but why the agent chose that path and whether the choice was within policy. This is where safety evaluation and security telemetry meet. The practitioner conclusion is that agent oversight has to connect behavioural explanation to security accountability.

Shift-left failed in traditional software because development and security stayed organisationally separate. The article correctly notes that AI offers a chance to avoid repeating that pattern if safety researchers and security teams work from the same operating assumptions. That does not mean the problem becomes easy. It means the governance model can be designed earlier around shared evidence, shared language, and shared control ownership.

The identity discipline now has to stretch across human, machine, and autonomous actors without changing its core logic. The same lifecycle questions appear in different forms, but autonomous agents raise the bar because their actions are not fully knowable at provisioning time. That makes access review, monitoring, and incident response less about fixed identities and more about governed runtime behaviour. Practitioners should expect their identity programme to absorb AI agent oversight rather than bolt it on separately.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.

What this signals

Decision-path assurance: AI agent governance will increasingly depend on the ability to explain not only what a system did, but why it chose that path. That shifts control design toward runtime evidence, shared review criteria, and a single operating model for both safety and security.

The programme implication is straightforward: if safety and security teams keep separate views of the same agent, organisations will miss the point where behaviour, policy, and accountability intersect. The effective control plane is the one that can inspect decisions, not just results.

For practitioners

• Define a joint safety-security control model Map which agent behaviours are evaluated for alignment, which are monitored for misuse, and which are escalated as security events. Keep those boundaries explicit so model risk and operational risk do not get handled by disconnected teams.
• Treat agent decision paths as governed evidence Capture the sequence of prompts, tool calls, and action choices that led to a material outcome. That gives reviewers something stronger than a final result and makes post-incident analysis possible.
• Separate runtime discretion from fixed automation Inventory which AI systems can choose actions at runtime and which only execute predetermined workflows. Apply the stricter governance model to any system that can change its own action path or timing.
• Build shared escalation criteria for safety and security teams Agree in advance which agent behaviours trigger model review, security response, or both. The goal is a single operating picture, not separate queues that slow containment or obscure intent.

Key takeaways

• AI agent governance cannot be split cleanly between safety and security once the actor can make runtime decisions.
• The article frames secure by design for AI as a shared discipline, not a sequence of isolated controls.
• Practitioners should build one oversight model that connects decision paths, monitoring, and escalation for agentic systems.

Key terms

• AI Agent Security Risk Review: An AI agent security risk review is an internal assessment that tests what a deployed agent can actually be made to do, who owns it, and whether its behavior fits bank risk policy. It goes beyond documentation to examine live tool use, data access, and governance accountability before or after deployment.
• AI Safety: AI safety is the discipline of preventing an AI system from taking unintended or harmful actions on its own. It focuses on the behaviour the system generates, even when no external attacker is involved. For identity teams, safety is about limiting what the agent can do once it is already operating.
• Decision Path Assurance: Decision path assurance is the ability to explain and verify the sequence of choices an AI agent made before acting. It matters because security teams need evidence of why the agent selected a path, not only what the end result was.
• Runtime discretion: The ability of a system to make access and action decisions while a task is in progress, rather than only at provisioning time. For autonomous identities, this creates a moving governance target because the effective privilege set can change during the session.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.