Should organisations trust an AI model’s safety filters to stop malicious use?

Why AI Safety Filters Are Not a Security Boundary

Safety filters are useful, but they are only one layer of defence and they are easy to probe. A determined user can rephrase the request, split the task into smaller steps, open a new session, or route the same intent through a different interface until the model yields.

That means the real control objective is not “make the model refuse bad prompts”, but “prevent the organisation from executing unsafe requests even if the model is persuaded”. The governing layer has to sit outside the model, where policy can be enforced consistently.

For a broader control view, NHIMG’s AI Governance Guide is the right place to anchor approval logic, human oversight, and use-case registration before an AI system is allowed to act.

What Organisations Need to Control Outside the Model

The durable pattern is to separate model output from operational authority. Even when a model can generate convincing instructions, code, or approvals, it should not be able to directly trigger high-impact actions without policy checks, scoped tools, and explicit approval paths.

That usually means restricting what the application can do, not only what the model can say. If the system can send messages, move money, change records, call tools, or expose data, each of those actions needs external enforcement, logging, and, where appropriate, human confirmation.

This is why NHIMG’s AI Acceptable Use Policy Template matters here: it covers approved tools, human review, and prohibited uses, which are the practical controls that sit above model behaviour.

For runtime containment, Zero Trust for AI Agents captures the core operating principle well, verify the request and remove standing privilege so each action is checked before it is allowed to proceed.

How to Test Whether Your Safeguards Actually Work

The useful question is not whether the model usually refuses harmful prompts, but whether the surrounding system still blocks the action when the model is bypassed. Red-team the end-to-end workflow, not just the chat experience, and test whether a rewritten prompt, a fresh session, or a different tool path still hits the same policy gate.

Pay special attention to high-risk functions such as credential access, data export, code execution, administrative changes, and external side effects. If those can happen without a separate approval step, the organisation has delegated too much trust to the model layer.

NHIMG’s AI Model Security Guide is useful when you want to separate model weaknesses, such as evasion and adversarial prompting, from the security controls needed around inference and action handling.

Risk and Threat Considerations

When organisations trust safety filters too much, they create a brittle control that fails under repetition, obfuscation, or session resets. The result is not just policy violation, it is unsafe execution through a channel that was assumed to be constrained.

Failure mechanism: The attacker adapts the prompt or restarts the interaction until the model produces the desired output, then uses that output to drive an action path that was never independently enforced.

Impact: Harmful content generation, unauthorised actions, data exposure, and abuse of tools or downstream systems become possible even when the model appears to be “safeguarded”.

Practitioner Guidance

What to prioritise: Treat external policy enforcement as the primary control and safety filters as a supplementary signal. If the model can influence a high-impact workflow, the workflow must have its own approval, logging, and permission checks.

What to verify: Confirm that the system still blocks disallowed actions when the same request is reworded, split across turns, or retried in a new session. If the answer changes but the action control does not, the architecture is too dependent on model refusal.

Decision rule: If a model output can change state outside the chat, require a non-model control before execution. If it cannot, the filter may be useful for user experience, but it should never be treated as the last line of defence.

Practitioner takeaway: The safer design is to assume the model will be persuaded eventually, then ensure the organisation still refuses unsafe execution at the application and policy layers.

Related resources from NHI Mgmt Group

• How should organisations stop auto-sync from turning desktops into repositories of credentials?
• Should organisations use just-in-time access for AI model operations?
• Should healthcare teams use the same zero trust model for AI agents and service accounts?
• Should organisations rely on model safety features alone to stop prompt injection?