What breaks when legacy security controls are used for conversational AI?

Where legacy controls stop matching the AI control plane

Legacy controls were designed around stable objects, such as files, endpoints, web requests and logged-in user sessions. Conversational AI is different because the security boundary is the interaction itself: a prompt can shape a response, the response can trigger tools, and the tool output can become the next instruction. That means the control point is no longer just access to a system, but governance over a sequence of context-bearing actions.

This is why many familiar controls appear to work while leaving the actual AI workflow only partially governed. A DLP rule, WAF policy or session timeout may still function, but none of those controls inherently understand whether the model is being steered toward data exposure, prompt injection, unsafe tool use or an action that should never be automated. For a useful control model, the question is not only “who connected?” but “what did the conversation cause the system to do?”

That shift from static transactions to conversational state is easy to underestimate. It changes how organisations think about logging, approvals, separation of duties and blast radius, because the risk often sits in the handoff between natural language and downstream execution. Guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls remains useful for access control and audit, but conversational systems need those controls applied to the interaction flow, not just the underlying infrastructure.

Why intent and tool use break the legacy model

Conversational AI introduces two things that legacy controls usually do not reason about well: intent and delegation. Intent can be benign, ambiguous or malicious, and the system often has to infer it from natural language rather than a rigid transaction structure. Delegation is even more consequential, because the model may act through tools, connectors or APIs that carry real permissions and can cause real side effects.

Traditional security controls struggle here because they are oriented toward explicit operations. They can verify that a user authenticated, a request reached an application, or a session stayed alive, but they do not natively decide whether a prompt is trying to extract sensitive context, override policy, or use the model as a proxy to reach a protected action. That gap is exactly where conversational systems become harder to govern than ordinary web apps.

For that reason, identity and authorisation are still relevant, but they must be attached to the action being taken, not only the person or process that opened the chat. When an assistant can search, write, summarise, send, approve or call tools, the security question becomes whether the specific action is allowed in that context, with that data, and with that scope. A useful reference point is RFC 8693: OAuth 2.0 Token Exchange, because delegation and on-behalf-of flows are closer to the control problem than a simple login session is.

In practice, this is where conversational systems often expose a hidden policy mismatch. The organisation thinks it is protecting a chat interface, while the model is actually operating as a decision and execution layer. Once tool access is involved, the relevant control question is whether every delegated action has a narrowly defined authority, a bounded context and a traceable approval path.

What production teams should redesign first

Teams usually get the most value by redesigning controls around the workflow, not the interface. That means defining which prompts, conversations, tool calls and outputs are merely informational, which are advisory, and which may initiate downstream action. It also means deciding where human review is mandatory, because not every AI action should be treated as equivalent to a user click.

Controls should then be rebuilt around observability and containment. Logs need to capture prompt, response, tool invocation, retrieved context and resulting action as one chain. Approval rules need to consider whether a model can reach regulated data, whether a connector can write back into business systems, and whether a single bad prompt can create an outsized blast radius. The more an assistant can act, the more its identity and permissions need to look like a governed workload, not an informal convenience feature.

NHIMG’s Agentic AI Security Guide is useful here because it frames the problem as inputs, memory, tools, orchestration and identity working together. For teams selecting controls, Agentic AI Security Policy Template is a practical reminder that registration, ownership, monitoring and retirement matter as much as the chat experience itself. For the infrastructure side, AI Infrastructure Workload Identity Guide helps anchor the control discussion in the identities behind the AI platform and its connected services.

Practitioner Guidance

What to verify: Verify that your controls can answer three questions for any AI interaction: what context entered the model, what action the model could trigger, and what evidence ties the action back to the conversation. If the answer stops at “the user was authenticated,” the control design is not yet sufficient.

Decision rule: If a model can only recommend, treat the control problem as advisory output management; if it can retrieve, write, send or approve, treat it as delegated authority and enforce explicit scope limits, review thresholds and auditability.

Common mistake: Do not assume a familiar enterprise control automatically applies because the AI feature sits inside an existing product. Conversational systems often inherit the old control label while bypassing the old control logic.

What good looks like: The organisation can show which prompts are allowed to reach tools, which outputs are blocked from action, and which conversations required human intervention before any side effect occurred.

Practitioner takeaway: The real break is not just that legacy controls miss prompts, it is that they miss delegated intent, so the control model must move from session governance to interaction governance.

Related resources from NHI Mgmt Group

• What breaks when security teams depend on legacy email controls to stop modern AI-generated phishing?
• What breaks when security programmes rely on legacy controls in an AI-driven threat environment?
• What is MCP in the context of AI security?
• What are the security risks associated with AI agents?