Most LLM incidents are not exotic model attacks: they come from ordinary deployment mistakes such as leaked API keys, exposed inference servers and models given more access than any user would get. This list ranks the ten mistakes that appear most often in the NHI Mgmt Group breach database, weighted by impact. It applies whether you call a hosted model API, run open-weight models yourself or build retrieval and tool-using applications on top. Model-level risks are covered in the OWASP Top 10 for LLM Applications Guide; this page focuses on what goes wrong when an LLM reaches production. It is written for platform, application security and AI engineering teams.
The Top 10 at a glance
The order reflects how often each mistake led to real compromise or data exposure in incidents we have analysed, then how much damage it allows. Each item maps to the OWASP Top 10 for LLM Applications 2025 and to MITRE ATLAS or ATT&CK techniques.
| # | Mistake | Typical outcome | OWASP LLM 2025 |
|---|---|---|---|
| 1 | Static, over-shared LLM API keys | LLMjacking, model access sold to others | LLM10 |
| 2 | Exposed self-hosted model servers and AI tooling | Remote code execution, secret theft | LLM03, LLM10 |
| 3 | Untrusted content meeting sensitive data | Zero-click data exfiltration | LLM01, LLM02 |
| 4 | Excessive agency | Destructive or unauthorised actions | LLM06 |
| 5 | Unvetted models and AI packages | Backdoored models, stolen credentials | LLM03, LLM04 |
| 6 | Retrieval that ignores permissions | Users see documents they should not | LLM08, LLM02 |
| 7 | Secrets and rules in the system prompt | Credential and logic disclosure | LLM07 |
| 8 | Trusting model output | Injection downstream, binding promises | LLM05, LLM09 |
| 9 | No spend or rate limits | Denial of wallet | LLM10 |
| 10 | No inventory, logging or testing | Shadow AI and incidents nobody sees | LLM01, LLM10 |
1. Static, over-shared LLM API keys
The issue: provider API keys committed to code, pasted into front-end apps, shared across teams and never rotated.
Why it matters: a model key is a paid, privileged non-human identity. A developer's commit exposed private Grok models in the xAI API key leak, and Storm-2139 used stolen customer keys to sell Azure OpenAI access. The wider pattern is in our LLMjacking analysis.
What to do: keep keys server-side only, one key per application and environment, prefer workload identity federation where the provider supports it, scan repositories for keys and rotate on a schedule. See the LLMjacking Guide.
2. Exposed self-hosted model servers and AI tooling
The issue: inference servers, orchestration tools, AI gateways and their databases reachable from the internet with no authentication or default keys.
Why it matters: attackers scan for them. Langflow's missing authentication flaw (CVE-2025-3248, in the CISA KEV catalogue) was used to dump AI server secrets and build a botnet, ShadowRay hit exposed Ray clusters, DeepSeek left an open ClickHouse database with chat logs, and a LiteLLM auth bypass and default master keys exposed gateway credentials.
What to do: put inference endpoints behind authentication and private networking, change default master keys, patch AI tooling like any internet-facing service and run models in hardened containers or VMs with allowlist firewalls, as the 2024 joint agency guidance on deploying AI systems recommends. See the Self-Hosted LLM Security Guide.
3. Untrusted content meeting sensitive data
The issue: the same assistant reads email, web pages or form submissions and also has access to mailboxes, CRM records or files.
Why it matters: indirect prompt injection (ATLAS AML.T0051) turns attacker text into instructions. EchoLeak showed zero-click data leakage from Microsoft 365 Copilot, and ForcedLeak used a web form and an expired domain to exfiltrate CRM data.
What to do: segregate and label external content, block rendering of external images and links in output, restrict egress to allowlisted domains, and treat any session that has read untrusted content as untrusted. See the Prompt Injection Guide.
4. Excessive agency
The issue: LLM applications given broad tools (shell, delete, send, pay) and credentials with more permissions than the task needs, acting without approval. OWASP LLM06 calls this excessive functionality, permissions and autonomy.
Why it matters: once injected or simply wrong, the model acts with whatever it holds. In the Amazon Q Developer extension compromise, an over-scoped token let an attacker ship a data-wiping prompt to an AI coding agent.
What to do: minimise tools and their functions, use per-user delegated credentials rather than one service account, enforce authorisation in downstream systems rather than in the prompt, and require human approval for high-impact actions. See the Agentic AI Security Guide.
5. Unvetted models and AI packages
The issue: downloading models, LoRA adapters and AI libraries from public hubs without checking provenance, and loading pickle-based weights that can run code.
Why it matters: the AI supply chain (ATLAS AML.T0010) is a credential target. The LiteLLM PyPI breach exposed users' LLM API keys through a poisoned package.
What to do: keep an approved model registry, verify hashes and signatures, prefer safetensors over pickle formats, scan imported models in an isolated zone first, and pin package versions. See the Model Supply Chain Guide.
6. Retrieval that ignores permissions
The issue: a RAG pipeline indexes SharePoint, Confluence or tickets with a service account, then answers every user from the whole index.
Why it matters: the assistant becomes a way round access controls: anyone can ask for salary files or board papers the index can see (OWASP LLM08).
What to do: carry source permissions into the vector store, filter retrieval by the requesting user's identity, partition indexes by tenant and sensitivity, and log which documents each answer used. See the Permission-Aware RAG Guide.
7. Secrets and rules in the system prompt
The issue: connection strings, API keys, internal role names or "never do X" rules placed in the system prompt, and confidential data pasted into public AI tools by staff.
Why it matters: system prompts can be extracted (ATLAS AML.T0056), so anything in them should be treated as public. Staff pasting code into a public tool led to the Samsung ChatGPT leak.
What to do: keep secrets out of prompts, enforce rules outside the model (OWASP LLM07), and give staff an approved tool with data controls plus a clear acceptable use policy.
8. Trusting model output
The issue: model output passed straight into SQL, shell commands, HTML or business decisions.
Why it matters: output handling flaws (LLM05) give classic injection and XSS, and confabulated answers (NIST AI 600-1) become commitments, as the Chevrolet dealer chatbot offering a $1 Tahoe showed.
What to do: treat output as untrusted user input: encode it for its context, use parameterised queries, validate against a schema and keep humans in the loop for anything binding.
9. No spend or rate limits
The issue: no per-user quotas, token limits or budget alerts on model usage.
Why it matters: stolen keys and abusive users run up bills quickly (ATLAS AML.T0034 Cost Harvesting, ATT&CK T1496.004 Cloud Service Hijacking), and OWASP LLM10 names denial of wallet explicitly.
What to do: route traffic through a gateway with per-key rate limits, input size limits, timeouts and hard budget caps, with alerts on spikes. See the AI Gateway and LLM Gateway Guide.
10. No inventory, logging or testing
The issue: nobody knows which models, keys and AI apps are in use, prompts and tool calls are not logged, and nothing is red-teamed before release.
Why it matters: every mistake above goes unnoticed, and incident response starts without evidence.
What to do: keep a central model and key inventory, log prompts, responses, tool calls and errors with user identity (minding privacy), alert on anomalies and run adversarial testing before each major release. See the LLM Observability Guide and the Shadow AI Discovery Guide.
Mapping to OWASP
The table maps each mistake to the OWASP Top 10 for LLM Applications 2025, the NIST AI 600-1 Generative AI Profile risk it most affects, and MITRE ATLAS or ATT&CK techniques. See the NIST AI 600-1 Generative AI Profile Guide.
| Mistake | OWASP LLM 2025 | NIST AI 600-1 risk | ATLAS / ATT&CK |
|---|---|---|---|
| 1. Static API keys | LLM10 | Information Security | T1552.001, AML.T0034 |
| 2. Exposed servers | LLM03, LLM10 | Information Security | T1190 |
| 3. Untrusted content | LLM01, LLM02 | Information Security; Data Privacy | AML.T0051, AML.T0057 |
| 4. Excessive agency | LLM06 | Human-AI Configuration | AML.T0053 |
| 5. Unvetted models | LLM03, LLM04 | Value Chain and Component Integration | AML.T0010 |
| 6. Permissionless retrieval | LLM08, LLM02 | Data Privacy | AML.T0057 |
| 7. Secrets in prompts | LLM07 | Information Security | AML.T0056 |
| 8. Trusting output | LLM05, LLM09 | Confabulation | AML.T0051 |
| 9. No spend limits | LLM10 | Information Security | AML.T0034, T1496.004 |
| 10. No inventory or logging | LLM10 | Information Integrity | AML.T0024 |
Primary sources: OWASP: Top 10 for LLM Applications 2025, NIST: AI 600-1 Generative AI Profile, NSA, CISA and partners: Deploying AI Systems Securely, MITRE ATLAS: LLM Prompt Injection and CISA: Known Exploited Vulnerabilities Catalog.
Where to start
If you can do only three things this quarter, fix the mistakes behind most incidents:
- Inventory and lock down keys (mistakes 1 and 10): list every model provider key, move them server-side behind a gateway, rotate anything ever committed and add secret scanning to CI.
- Close exposed AI infrastructure (mistake 2): scan for internet-reachable inference servers, orchestration tools and vector databases, put them behind authentication and patch KEV-listed flaws first.
- Separate reading from acting (mistakes 3 and 4): remove tools and permissions that applications do not need, and require approval for any action triggered after the model has read external content.
Measure progress with four numbers: model keys older than 90 days, internet-exposed AI endpoints (target zero), the share of LLM applications with per-user rate limits and budget caps, and the share released only after adversarial testing. Small teams using one hosted provider should start with key hygiene and spend caps; large organisations running their own models need a gateway, a model registry and permission-aware retrieval before scaling further. To judge where you stand overall, use the AI Security Maturity Model.
Related NHI Mgmt Group resources: LLM Security Guide · RAG Security Guide · AI Model Security Guide · Top 10 Secrets Management Mistakes