AI safety and AI security are not the same governance problem

At a glance

What this is: This is a governance analysis of why AI safety and AI security are distinct risks, and why enterprises need separate controls for each.

Why it matters: It matters because IAM, PAM, and broader identity programmes now intersect with AI tools, agents, and developer workflows that can be misused, over-trusted, or steered off-path.

👉 Read Knostic's analysis of where AI safety and AI security diverge

Context

AI safety and AI security are often blurred in enterprise planning, but they describe different failure modes. Safety is about harmful outcomes from a system doing what it was designed to do, while security is about an attacker bending the system away from intended use. That distinction affects governance, control ownership, and production readiness, especially once AI tools are embedded in identity-sensitive workflows.

The identity intersection is real when AI systems can access repositories, environment variables, SaaS tools, or delegated credentials. In that setting, access scope, logging, and review become part of AI governance, not just IT hygiene. For a useful companion view on how AI controls map into security programmes, see the NIST Cybersecurity Framework 2.0 and NHIMG's Top 10 NHI Issues.

Key questions

Q: How should security teams govern AI assistants that can access enterprise tools?

A: Treat AI assistants as privileged systems, not as simple interfaces. Limit the repositories, secrets, and SaaS actions they can reach, log every tool call, and require explicit approval for any change in access scope. If the assistant can act on behalf of users or services, it needs identity controls, reviewable permissions, and incident response coverage.

Q: Why do AI systems create both safety and security risk?

A: AI systems can cause harm even when they behave as designed, which is a safety issue, and they can also be manipulated by attackers, which is a security issue. Enterprises need both lenses because the controls, owners, and evidence differ. Conflating them usually leaves governance gaps that no single team can fully own.

Q: What do organisations get wrong about AI security governance?

A: They often assume model hardening is enough, while the real exposure sits in tool access, delegated credentials, logging, and approval paths. If an AI system can reach sensitive data or execute actions, governance must focus on who can change that access and how misuse is detected. Otherwise the deployment is larger than the control model.

Q: Who should be accountable for AI safety and AI security decisions?

A: Accountability should sit with a clearly named owner who can coordinate security, legal, product, and operational decisions. Safety and security should not be split into disconnected silos because the same AI system can create harm, compliance risk, and adversarial exposure at once. The accountable owner must control change approval and escalation.

Technical breakdown

AI safety vs. AI security: why the failure mode matters

AI safety asks whether a system causes harm even when it behaves as designed. AI security asks whether an adversary can manipulate, steal, or weaponize the system by altering inputs, tools, training data, or execution context. The two can overlap, but they are not interchangeable. A model can be technically secure and still produce discriminatory, dangerous, or harmful outcomes. It can also be safety-aligned in theory yet easy to exploit in practice. Practitioners need separate control objectives because the threat model, evidence, and accountability differ.

Practical implication: write distinct safety and security acceptance criteria before deploying AI into production.

Why MCP servers and coding assistants expand the AI attack surface

MCP servers, IDE extensions, and coding assistants extend AI from text generation into action. Once a system can read repositories, inspect environment variables, call tools, or trigger workflows, prompt injection and malicious integration become execution paths, not just content risks. The attack surface includes the model, the wrapper, the connector, and the surrounding trust assumptions. That is why AI security is increasingly a control-plane issue: the toolchain determines whether a prompt can become a command, a lookup, or a data leak.

Practical implication: restrict tool and data access for AI assistants to the minimum required by each workflow.

The governance gap: who owns AI change control and access scope?

Many organisations assign AI safety to legal or policy teams and AI security to technical teams, then leave product and engineering to move faster than either can govern. That produces unclear RACI, weak exception handling, and inconsistent production gates. When AI systems can affect employees, customers, or regulated data, governance must define who can change model behaviour, tool permissions, logging, and escalation paths. Without that, the organisation is relying on informal judgment rather than accountable control.

Practical implication: assign a single accountable owner for AI access scope, change approval, and production sign-off.

Threat narrative

Attacker objective: The attacker wants to turn trusted AI tooling into a path for credential theft, code tampering, or broader environment compromise.

1. Entry begins when a compromised coding assistant, malicious extension, or prompt injection reaches an AI-enabled workflow with tool access.
2. Escalation occurs when the AI system is induced to read, reveal, or act on data and credentials beyond the user’s intended context.
3. Impact follows when the manipulated system exfiltrates secrets, runs destructive commands, or introduces backdoors into code and infrastructure.

NHI Mgmt Group analysis

AI safety and AI security should be treated as separate governance domains. Safety evaluates whether a system can cause harm while still operating as intended. Security evaluates whether an adversary can bend that system off its intended path. Enterprises that collapse the two usually misassign ownership and misjudge readiness. The practical consequence is weak accountability across legal, product, and security functions.

AI tooling has become an identity and access problem, not just a model problem. Once coding assistants, MCP servers, and agentic workflows can touch repositories, environment variables, and SaaS tools, the question becomes who or what is authorised to act. That creates a genuine intersection with IAM, PAM, and NHI governance, because delegated access and tool permissions now define the blast radius. Practitioners should treat AI connectors as privileged integration points.

AI governance debt: the gap between what an AI can do and what the organisation has formally approved. This article exposes a familiar enterprise failure mode where deployment outpaces control design. If teams cannot describe who approved tool access, what logging exists, and what escalation path stops misuse, the issue is not only technical risk but governance debt. The conclusion is simple: define the approved operating envelope before autonomy expands further.

Security leaders should stop treating AI incidents as purely application-layer events. Prompt injection, malicious MCP servers, and compromised assistants can become identity abuse channels when the system inherits broad credentials or session context. That means access boundaries, secrets handling, and session scope matter as much as prompt filters. The field should expect more incidents where the root cause is over-trusted delegation rather than model failure alone.

Production readiness for AI now depends on control evidence, not enthusiasm. Enterprises should require proof that safety constraints, security controls, and identity checks work together under realistic workflows. If the AI can access sensitive systems, then logging, least privilege, and exception handling must be demonstrable before scale-up. The practitioner lesson is to make evidence, not intent, the deployment threshold.

What this signals

AI governance teams should expect the control conversation to move from model behaviour alone to delegated access, tool permissions, and session scope. That shift matters because once an AI system can touch enterprise systems, the control surface looks much closer to privileged access management than to a pure software review. The practical implication is that AI readiness now depends on identity evidence, not just policy language.

Governance debt: the gap between what an AI system is allowed to do in practice and what the organisation can prove it approved. That gap widens quickly when coding assistants and MCP-connected tools inherit broad access from users or services. Enterprises that cannot evidence least privilege for AI tooling should assume their deployment will outpace their governance model.

For teams managing identity-heavy environments, AI controls should be assessed alongside secrets management, logging, and exception handling. If a model, agent, or assistant can reach credentials, then the issue is no longer only AI risk, it is identity risk with AI acceleration. The next phase of maturity is to connect AI access decisions to existing identity review and incident processes.

For practitioners

• Separate safety gates from security gates Define distinct approval criteria for harmful-output risk and adversarial misuse risk before any AI system reaches production. Tie the two gates to different owners, test cases, and escalation paths.
• Restrict AI tool access by workflow Limit MCP servers, coding assistants, and connected tools to the smallest set of repositories, secrets, and actions required for each use case. Review those permissions like privileged access, not like ordinary application settings.
• Treat AI connectors as privileged integrations Log every tool invocation, credential use, and data retrieval event for systems that can act on behalf of users or services. Map those integrations to identity controls, access reviews, and incident response.
• Create an AI governance owner with change authority Assign a single accountable leader for tool access, model change control, and deployment sign-off so that security, legal, and product teams are not operating from conflicting assumptions.

Key takeaways

• AI safety and AI security describe different failure modes, so they need different owners, controls, and production gates.
• When AI tools can access repositories, secrets, or SaaS systems, the real issue becomes delegated privilege and tool governance.
• Enterprises should treat AI readiness as a control-evidence problem, not as a branding exercise around responsible innovation.

Key terms

• AI Safety: AI safety is the discipline of preventing harmful outcomes when an AI system behaves as designed. It focuses on alignment, unintended consequences, and downstream impact on people, rights, or operations, even when no attacker is present. In practice, it requires policy, testing, and governance evidence beyond traditional security controls.
• AI Security: AI security is the protection of AI systems against adversarial manipulation, theft, misuse, and weaponization. It covers prompt injection, compromised tools, poisoned data, malicious integrations, and abuse of infrastructure around the model. The goal is to preserve intended behavior under attack, not to judge whether the behavior itself is desirable.
• MCP Server: An MCP server is a tool connector that lets an AI system interact with external data sources and services through the Model Context Protocol. Because it can expose actions, context, or data to the model, it becomes part of the trust boundary and must be governed like a privileged integration.
• Governance Debt: Governance debt is the gap between what an organisation allows a technology to do and what it can formally prove it has approved, monitored, and controlled. In AI programmes, it appears when autonomy, access, and tool reach expand faster than ownership, evidence, and review processes can keep up.

What's in the full article

Knostic's full research covers the operational detail this post intentionally leaves for the source:

• Specific examples of where AI safety and AI security diverge in enterprise governance decisions
• Practical discussion of how coding assistants, MCP servers, and agentic workflows expand the attack surface
• Why the organisation's RACI often fails when legal, product, security, and data teams split ownership
• Implementation context for balancing model behaviour, delegated access, and accountability

👉 The full Knostic article expands on governance ownership, AI risk boundaries, and the practical tension between safety and security.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It gives practitioners a practical way to connect identity controls to the systems that now power AI-enabled workflows.