Why do deployed LLMs create risk even when the underlying infrastructure is secure?

Why This Matters for Security Teams

Secure infrastructure reduces exposure at the platform layer, but deployed LLMs introduce a second attack surface: the model, the prompts, the retrieval layer, and any tools the model can call. That means a cloud account can be well protected while the application still leaks data, follows hostile instructions, or produces unsafe actions. Guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 both point to the same operational truth: AI risk is not eliminated by securing servers, identities, and networks alone.

Security teams often assume the model will only answer questions, but deployed systems increasingly read documents, fetch context, invoke APIs, and trigger workflows. Each of those steps can be manipulated if the application fails to separate trusted instructions from untrusted content. The result is not just incorrect output. It can become credential exposure, policy bypass, data exfiltration, or unsafe automation that appears legitimate because it came from an approved system. In practice, many security teams encounter LLM misuse only after sensitive content has already been exposed through normal application behaviour, rather than through intentional testing.

How It Works in Practice

A secure deployment starts with recognising that the infrastructure boundary and the model boundary are different. Cloud controls protect compute, storage, and identity, but they do not automatically constrain how the model interprets input or which data it will surface in response. Practitioners need layered controls across prompt handling, retrieval, tool use, logging, and post-deployment monitoring.

• Separate system instructions, user input, and retrieved content so hostile text cannot override trusted policy.
• Filter and label retrieved documents, because poisoned content in a knowledge base can become an indirect attack path.
• Constrain tool access with least privilege, short-lived credentials, and explicit approval gates for risky actions.
• Test for prompt injection, jailbreak patterns, sensitive data leakage, and unsafe tool invocation before release.
• Log prompts, outputs, tool calls, and refusals so security teams can detect drift or abuse after deployment.

The NIST AI 600-1 Generative AI Profile is useful here because it translates general AI governance into controls for generative systems, including data handling, content integrity, and monitoring. For deeper threat modelling, the MITRE ATLAS adversarial AI threat matrix helps teams map concrete attack techniques such as prompt injection, model evasion, and supply chain tampering. These controls tend to break down in highly autonomous agent environments because the model can chain multiple tool calls faster than manual oversight can intervene.

Common Variations and Edge Cases

Tighter LLM controls often increase latency, workflow friction, and engineering overhead, requiring organisations to balance user experience against containment and auditability. That tradeoff becomes sharper when the model is embedded in customer support, software delivery, or internal copilots where speed is valued more than scrutiny.

Current guidance suggests there is no universal standard for how much autonomy is acceptable, especially for agentic systems that can act on behalf of users. Some deployments only need content safety and leakage prevention, while others need approval workflows, transaction limits, and full provenance tracking. The right answer depends on whether the system merely drafts text or can change records, send messages, or execute code. The CSA MAESTRO agentic AI threat modeling framework is particularly relevant where tools and workflows are combined, because it forces teams to model the agent’s decision path rather than treating the model as a passive text generator.

Another edge case is retrieval-augmented generation. RAG can reduce hallucination, but it can also expand the attack surface if source documents are untrusted, stale, or overly permissive. The safest pattern is to treat every retrieved item as potentially adversarial until validated. This guidance also applies when multiple identities are involved, including human users, service accounts, and non-human identities that grant the model access to downstream systems.

Related resources from NHI Mgmt Group

• Why do APIs create identity risk even when the application code is secure?
• Why do AI model servers create NHI governance risk even when deployed locally?
• Why does Google Drive create PCI risk even when the platform is secure?
• Why do payment page scripts create compliance risk even when the application looks secure?