Who is accountable when AI applications ship without runtime controls or testing?

Accountability sits with the organisation deploying the system, not the model provider alone. Security, application, compliance, and risk owners should define the control requirements before release, then verify them through testing and monitoring. If the system can act autonomously, governance must cover access, actions, logging, and escalation paths across the full runtime lifecycle.

Why This Matters for Security Teams

Runtime controls are the difference between a system that is merely deployed and one that is actually governed. When AI applications ship without tested controls, the organisation inherits the risk even if the model or platform came from a third party. That includes overbroad access, unlogged actions, weak escalation paths, and secrets exposure. NIST guidance on access control and continuous monitoring in NIST SP 800-53 Rev 5 Security and Privacy Controls is relevant here because the issue is not model quality alone, but whether the deployed system can be constrained and observed in production.

For AI-specific risk, the better question is whether the application can act outside expected boundaries once it is live. NHIMG research on Ultimate Guide to NHIs — Standards frames that as an identity and control problem, not just a software delivery problem. If an autonomous workflow can call tools, retrieve secrets, or trigger downstream actions, then release decisions must include runtime authorization, telemetry, and revocation. In practice, many security teams encounter those gaps only after an incident review reveals that “go-live” meant code was deployed, not that it was actually controlled.

How It Works in Practice

Accountability should be assigned across three layers: the business owner who approves the use case, the engineering owner who implements the control plane, and the security owner who verifies the controls before production. For AI applications, that means defining what the system may access, what it may do, and what evidence will prove it stayed within bounds. A control checklist is not enough unless it is backed by runtime enforcement and testable telemetry.

In operational terms, the system should be evaluated at request time, not only during build or code review. That includes identity-bound access for the workload, least privilege for tool calls, short-lived credentials, and logs that capture the action, context, and outcome. This aligns with the control intent behind NIST SP 800-53 Rev 5 Security and Privacy Controls, but the implementation pattern is increasingly shaped by NHI governance guidance such as DeepSeek breach, which illustrates how exposed credentials and weak runtime boundaries amplify downstream harm.

• Test whether the AI system can only invoke approved tools and endpoints.
• Verify that credentials are ephemeral and revoked after task completion.
• Confirm that logs capture prompts, actions, policy decisions, and escalation events.
• Run abuse-case tests for tool chaining, privilege escalation, and data exfiltration.

Where possible, teams should define runtime gates in policy-as-code so that authorization reflects the current task, user, data sensitivity, and environment. These controls tend to break down in loosely governed multi-agent systems because the action path is dynamic and can change faster than static approval workflows.

Common Variations and Edge Cases

Tighter runtime control often increases delivery overhead, requiring organisations to balance speed against assurance. That tradeoff is real: a low-risk internal assistant does not need the same control depth as an agent that can move tickets, send email, or call production APIs. Current guidance suggests calibrating controls to execution authority, not to the AI label alone.

There is also no universal standard for this yet. Some teams treat runtime testing as a security sign-off gate, while others fold it into continuous verification after release. The practical difference is whether failures are caught before the system gets credentials and network reach, or only after monitoring detects misuse. For regulated environments, accountability should be explicit in policy and contracts, but the organisation still owns the control failure if the system is deployed without effective checks.

NHIMG research on The State of Secrets in AppSec is a useful reminder that governance gaps often show up through secrets handling, not just model behaviour. If runtime controls are absent, a compromised secret, permissive token, or untested escalation path can turn a routine AI workflow into an incident. The main edge case is vendor-hosted or embedded AI where the provider operates part of the stack, because accountability still depends on the deploying organisation proving that control expectations were specified, tested, and monitored.

Related resources from NHI Mgmt Group

• When should organisations add runtime controls to AI applications?
• Why do AI agents and tool-connected LLMs need runtime controls as well as testing?
• What makes agentic AI an NHI governance issue?
• Why is NHI governance critical in the age of AI attacks?