AI security is moving from model testing to runtime defense

At a glance

What this is: This is an interview exploring why AI security is shifting from model testing toward runtime defense as enterprises deploy AI, including agentic AI systems, at scale.

Why it matters: It matters because IAM and security teams now have to treat AI deployments as governed runtime systems, not just pre-release models, with implications for access, oversight, and control boundaries.

Context

AI security is the set of controls that protect models, applications, data, and the actions those systems can take after deployment. The article argues that the field has outgrown a narrow focus on pre-release testing because production AI now behaves as part of the operational environment, not just a development artefact.

For identity and access practitioners, the important shift is that build-time assurance and runtime governance now overlap. Once AI systems can call tools, interact with enterprise data, or operate as agentic systems, the question becomes how access, privilege, and monitoring apply while the system is live.

Key questions

Q: What breaks when AI risk assessment stops at model testing?

A: You miss the non-technical risks that usually decide whether the system is safe to deploy. A model can pass bias or security checks and still fail because its data is sensitive, its outputs are opaque, its permissions are excessive, or its operating context changes after approval.

Q: Why do AI governance programs need runtime enforcement for agentic workloads?

A: Agentic workloads can take actions autonomously, which makes delayed oversight too late to prevent exposure. If governance only monitors after the fact, prompts may already have leaked data, costs may already be spent, and unauthorized tool use may already be executed. Runtime enforcement matters because it blocks unsafe behavior before the model or agent acts.

Q: How should organisations combine build-time and runtime AI controls?

A: Treat build-time testing and runtime defense as complementary controls. Build-time work finds weaknesses before release, while runtime controls limit harm when behaviour changes in production. The two should share ownership, escalation paths, and visibility so findings from testing inform live monitoring and production incidents feed back into assurance.

Q: How should security teams govern agentic AI as it moves into production?

A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature. That means assigning ownership, scoping permissions tightly, logging every tool action, and revoking access on a defined lifecycle. Production rollout should require clear approval points for high-risk actions and continuous monitoring for drift.

Technical breakdown

Why model testing is no longer enough for AI security

Model testing looks for adversarial behaviour before release, but it does not describe what happens when a model is embedded in an operational workflow. Runtime defense covers the live layer where prompts, outputs, policies, tools, and surrounding systems interact. That is where an AI application can be exposed even if the model passed earlier testing. The article’s core point is that enterprise AI no longer sits inside a clean lab environment. It sits inside business processes, so security has to evaluate what the system can do after deployment, not only what it can predict in a test harness.

Practical implication: Treat pre-release testing as necessary but incomplete, and govern AI systems as live production services once they can affect enterprise workflows.

What runtime defense means for agentic AI systems

Agentic AI adds a sharper identity problem because the system can choose actions and invoke tools during execution. That creates a security boundary around not just the model response, but the permissions attached to the agent’s runtime context. In practice, the control question shifts from whether the model is safe in isolation to whether the agent can reach the wrong data, call the wrong tool, or execute outside its intended scope. The article points to agentic AI as part of the environment already being secured, which makes runtime oversight a governance requirement rather than an optional enhancement.

Practical implication: Define runtime permissions, tool access, and monitoring for agentic systems as explicitly as you define them for human or service identities.

How build-time and runtime controls fit together in AI security

Build-time controls such as red teaming and runtime controls such as defensive monitoring solve different problems. Build-time work finds weaknesses before release; runtime defense limits harm after deployment when conditions change, prompts vary, and integrations expand. The article frames this as a practical operating model for enterprise AI, not an academic one. That matters because many organisations still separate model assurance from operational security, even though the real risk sits in the gap between them. The stronger architecture is one where each layer informs the other, especially when AI applications are deployed across a broad innovation programme.

Practical implication: Use separate control objectives for pre-deployment assurance and in-production containment, then connect them through shared ownership and escalation paths.

Breaches seen in the wild

• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime defense is now the control plane that matters most for enterprise AI. Once AI moves into production, model quality alone stops being a sufficient security signal. The operational question becomes what the system can access, invoke, or influence while it is live, especially in environments where AI is wired into business processes. Security leaders should treat runtime behaviour as the primary risk surface, because that is where safe design meets messy reality.

Agentic AI collapses the boundary between model assurance and access governance. An agent that can select actions and invoke tools is no longer just a model under evaluation. It becomes a governed actor with a runtime permission boundary, and that changes how access, monitoring, and containment must be reasoned about. The right unit of analysis is no longer the model in isolation, but the live system that combines model, tools, and execution context.

Build-time testing and runtime monitoring solve different identity problems. Red teaming is about finding weaknesses before release, while runtime defense is about limiting blast radius after deployment. Organisations that treat these as substitutes will miss the gap where production AI actually fails. The practitioner takeaway is that AI security programmes need both, but runtime governance is the layer that determines whether a bad decision becomes a live incident.

AI security is converging with broader identity governance, not standing apart from it. As AI applications are embedded across enterprise environments, they inherit the same governance questions that have long applied to service accounts, APIs, and other non-human identities. That convergence means security teams should stop treating AI protection as a siloed specialist topic. The programme that survives will be the one that can govern model behaviour, tool access, and accountability together.

Runtime exposure is the named concept that now defines the category. The article’s real insight is that the risk surface has moved from static model artifacts to live execution conditions. Runtime exposure means the operational state in which a model, agent, or AI application can be manipulated, misused, or over-extended after deployment. Practitioners should use that lens when deciding where AI security ownership begins and ends.

What this signals

Runtime exposure: The useful framing for AI security has shifted from whether a model can be fooled to how a deployed system can be manipulated after it is already trusted. That change matters because security controls must now cover live behaviour, not just pre-release evaluation.

Enterprises should expect AI security ownership to converge with identity governance as soon as systems can call tools, reach data, or act on behalf of a workflow. At that point, the control question is no longer only model integrity, but who can do what, under what runtime conditions, and with which containment boundaries.

For practitioners

• Map AI systems to runtime control boundaries Inventory which deployed AI applications can call tools, reach data, or influence downstream workflows, then document the control boundary for each one. This is the point where model assurance becomes operational governance.
• Separate build-time testing from live monitoring Keep red teaming, pre-release evaluation, and in-production telemetry as distinct controls with different owners and escalation paths. Runtime defense should answer how the system behaves after deployment, not just whether it passed testing.
• Define permissions for agentic systems explicitly Treat agentic AI as a runtime actor with scoped tool access, data access, and execution limits. The permission model should be written before the system is connected to production workflows, not after.
• Align security, data science, and business owners Establish a shared operating model so deployment decisions, monitoring thresholds, and incident escalation are not split across disconnected teams. The article shows that AI security only works when these groups stay coordinated.

Key takeaways

• AI security is no longer just a pre-release model-testing problem because production systems create new runtime exposure.
• Agentic AI makes access and execution boundaries part of the security model, not just the model evaluation process.
• The practical response is to govern build-time assurance and runtime defense together so live systems stay within defined boundaries.

Key terms

• Runtime ai security: Runtime AI security is the set of controls applied while a model is actively handling prompts and responses. It focuses on what the system sees, says, and does in production, rather than relying only on pre-deployment testing or static policy documentation.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions, including calling APIs, writing code, and orchestrating other agents, with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Build-Time Assurance: Build-time assurance is the pre-deployment process of testing and evaluating an AI system before it goes live. It helps identify weaknesses early, but it does not replace runtime controls because operational risk emerges once the system is connected to real workflows and data.
• Runtime exposure management: A security approach that prioritises what is actually exploitable in live environments rather than treating every finding as equally urgent. It combines runtime context, asset criticality, and control enforcement so teams can focus on exposure that changes production risk.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.