What do organisations get wrong about protecting AI assistants from malicious prompts?

Why This Matters for Security Teams

Most organisations still treat AI assistants as a content problem: block bad words, flag unsafe phrases, and assume the risk is contained. That misses the real issue. A malicious prompt becomes dangerous when it changes what the assistant can retrieve, disclose, or execute through connected tools, data sources, and delegated privileges. The control plane around the model matters more than the text itself, especially when assistants can chain actions across systems.

This is why incidents such as the DeepSeek breach and the Schneider Electric credentials breach matter to defenders beyond the headline. They show how exposed secrets, weak access boundaries, and overly trusting integrations turn model interaction into business compromise. NIST’s NIST Cybersecurity Framework 2.0 reinforces that identity, access, and governance are core security functions, not optional wrappers around AI.

In practice, many security teams encounter prompt abuse only after the assistant has already queried data, invoked a plugin, or leaked sensitive context through an authorised channel.

How It Works in Practice

Effective defence starts by separating natural-language input from authorisation. A prompt should never be treated as proof that the user, or the agent acting on their behalf, is allowed to access the requested system or data. Current guidance suggests designing AI assistants so that each tool call, retrieval step, or outbound action is evaluated at runtime against policy, not simply approved because the prompt was syntactically valid.

That means the model may interpret user intent, but the security layer decides whether that intent can be fulfilled. The practical pattern is to constrain the assistant with least privilege, explicit scopes, and short-lived credentials. NIST SP 800-53 Rev. 5 supports this approach through access control and audit requirements, while security design should also account for the way assistants blend retrieval, orchestration, and action into a single workflow.

• Use separate identities for the user, the assistant, and each tool or backend service.
• Issue just-in-time access for high-risk actions instead of long-lived standing credentials.
• Apply policy at the moment of retrieval or execution, not only at prompt submission.
• Log the prompt, the policy decision, and the downstream action as one traceable event.

NHIMG research on DeepSeek breach and The State of Secrets in AppSec shows how quickly exposed secrets and fragmented control can undermine these boundaries. This guidance breaks down in highly integrated environments where legacy plugins, shared service accounts, or unsegmented data planes let one assistant request inherit broad backend authority.

Common Variations and Edge Cases

Tighter prompt controls often increase operational friction, requiring organisations to balance user experience against the need to prevent unintended data access and tool misuse. That tradeoff becomes especially visible in assistants used for support, engineering, or finance, where users expect fast answers but the assistant may touch sensitive records.

There is no universal standard for this yet, but current practice is moving toward context-aware guardrails rather than one-size-fits-all prompt filters. Some environments only need retrieval restrictions and redaction. Others need step-up approval for execution, per-tool scopes, or human confirmation before the assistant can write, delete, or trigger external workflows. The right model depends on whether the assistant is read-only, advisory, or operationally empowered.

One frequent mistake is assuming a safe prompt equals a safe outcome. Prompt injection, indirect injection through documents, and malicious tool instructions can all bypass a purely content-based filter. Organisations should also watch for over-permissioned connectors, since a harmless-looking assistant may still inherit access to mail, storage, code, or ticketing systems. NIST’s control model and NHIMG guidance both point to the same operational lesson: constrain the action surface, not just the conversation.

Where AI assistants are connected to multiple business systems with shared tokens or poorly scoped service accounts, the control model breaks down because the model can be steered into actions that the security team never explicitly reviewed.

Related resources from NHI Mgmt Group

• What do security teams get wrong about protecting agentic AI from malicious prompts and poisoned context?
• What do organisations get wrong about filtering malicious prompts?
• What do organisations get wrong about shadow AI governance?
• What do organisations get wrong about AI agent inventory and visibility?