Why is controllability not the same as safety for AI systems?

Why This Matters for Security Teams

Controllability is useful, but it is not a safety guarantee. A system can comply with prompts, accept overrides, or expose guardrails and still generate harmful actions when those outputs are wired into deployment pipelines, ticketing systems, or tool-using agents. The real risk is not only what the model says, but what the surrounding system allows it to do.

This is why security teams need to separate model behaviour from execution authority. NIST’s NIST Cybersecurity Framework 2.0 treats governance and control as operational responsibilities, not model properties. NHIMG research on the DeepSeek breach shows how exposed credentials and accessible data can turn a technical weakness into a live incident. In AI environments, that pattern becomes more dangerous because an unsafe suggestion can be executed instantly if orchestration is too permissive.

Security teams often overestimate safety after seeing a model refuse a bad request, then discover the real exposure only when an agent, workflow, or connected secret makes the same request actionable in production.

How It Works in Practice

In practice, safety depends on a chain of controls around the model, not just the model itself. A controllable system can be redirected by policy, but the safest architectures prevent risky outputs from reaching execution paths unless they pass runtime checks. That means pre-execution policy, tool gating, approval steps, and short-lived credentials matter more than conversational compliance.

The operational model is simple: treat the AI as a decision-support component until policy permits action. If the system is an agent, the controls should verify intent, context, and authorization before any tool call, data retrieval, or write operation. Guidance from the NIST Cybersecurity Framework 2.0 and the Ultimate Guide to NHIs supports this separation between policy and execution.

• Use workload identity to prove which agent or service is acting, not just which user approved it.
• Issue just-in-time secrets and revoke them when the task ends.
• Evaluate policy at request time rather than relying on static role assumptions.
• Log tool use, prompt inputs, and downstream actions as separate control points.

For agentic systems, this aligns with current guidance from OWASP and CSA, where the focus is runtime authorization and constrained tool access, not trusting the model to self-police. The Schneider Electric credentials breach reinforces the same lesson: once credentials are available to the wrong path, control becomes execution. These controls tend to break down when legacy automation grants broad standing access because the model can then chain benign steps into high-impact actions.

Common Variations and Edge Cases

Tighter runtime controls often increase latency, integration effort, and approval overhead, so organisations must balance responsiveness against blast-radius reduction. That tradeoff is real, especially in high-volume environments where teams want agents to operate with minimal friction.

There is no universal standard for this yet, but current guidance suggests a few important distinctions. A chat model used for drafting is not the same as an agent that can send emails, rotate secrets, or trigger deployments. The former may only need content filtering and review, while the latter needs pre-execution authorization and time-bound credentials. In multi-agent systems, one model’s “safe” output can become another model’s instruction, which makes orchestration the real control plane.

This is also where static RBAC often falls short. Role-based access can define who is allowed to act, but it does not answer whether this specific action is safe right now. That is why best practice is evolving toward context-aware controls, policy-as-code, and zero standing privilege. The safest architectures assume that a model can still produce an unsafe suggestion, then make sure the suggestion cannot execute without passing an independent policy gate.

In practice, the hardest failures appear in environments where human approval is treated as a substitute for technical enforcement, because once approval becomes routine, the system stops distinguishing between low-risk suggestions and high-risk actions.

Related resources from NHI Mgmt Group

• What breaks when AI security systems are allowed to detect and remediate in the same workflow?
• Why do AI-enabled marketing systems increase privacy and security risk at the same time?
• Why do generative AI systems need simulation-based safety testing?
• How should teams govern access when AI agents and service accounts share the same business systems?