Most GenAI features ship with the model treated as a trusted component: it reads untrusted text, sees data the user should not, and its output flows straight into browsers, databases and tools. This template is a pre-launch security checklist for applications built on large language models, from a support chatbot to a retrieval-augmented (RAG) assistant to a feature that calls tools. It covers prompt injection, output handling, sensitive data leakage, RAG access control, tool permissions, cost and abuse limits, supply chain and logging, and maps every item to the OWASP Top 10 for LLM Applications 2025. It is written for product security engineers, application architects and reviewers who sign off a release. Replace the values in square brackets and approve it through your normal process.
How to use this template
- Replace text in square brackets, such as [Organisation], [AI security lead] and [50 requests per minute], with your own roles and values.
- Run the checklist at design review and again before launch; "must" items block launch, "should" items need an approved exception under section 8.
- Copy section 6 into your tracker with a Status column (Met, Not met, Exception, Not applicable). "Not applicable" is valid only where the feature genuinely lacks that component, for example no tools, no RAG.
- A check is met when the "Verify" test passes against the release candidate, ideally as an automated test in CI.
- For the background on each risk, read the OWASP Top 10 for LLM Applications Guide; this page is the sign-off list, not the explanation.
1. Purpose
This standard sets the minimum security controls an application feature that uses a large language model must have before it is exposed to users at [Organisation]. It applies the OWASP Top 10 for LLM Applications 2025, the NCSC Guidelines for secure AI system development (secure design, development, deployment, and operation and maintenance) and the NIST AI 600-1 Generative AI Profile to the release process. The governing principle comes from OWASP itself: treat the model as any other user, adopt a zero-trust approach to its output, and enforce security controls outside the model rather than in the prompt.
2. Scope
- Any [Organisation] product feature, internal tool or API that sends prompts to a hosted or self-hosted LLM and uses the response.
- Includes chatbots, copilots, summarisers, RAG assistants, classification and extraction pipelines, and features that let the model call tools or APIs.
- Autonomous agents with long-running goals are covered by [Agentic AI Security Policy]; see the Agentic AI Security Policy Template.
3. Roles and responsibilities
- [Product owner]: owns the feature's risk, documents intended use and the data it touches.
- [Engineering lead]: implements controls and the automated tests in section 6.
- [AI security lead]: reviews the threat model, runs or commissions adversarial testing and signs off launch.
- [Data protection officer]: approves personal data use, retention and provider terms.
- [Security operations]: receives LLM logs and alerts and runs the Prompt Injection Incident Response Playbook.
4. Definitions
- Direct prompt injection: a user's own input that changes the model's behaviour against the designer's intent.
- Indirect prompt injection: instructions hidden in content the model reads, such as web pages, emails, documents or retrieved chunks.
- Untrusted content: anything not written by [Organisation]'s developers: user input, retrieved documents, tool results, third-party data.
- Tool: any function, plugin, API or MCP server the model can ask the application to call.
- High-risk action: an action that sends data externally, changes records, spends money, or cannot be undone.
5. Risk tiering
Scale the checklist to what the feature can do. Assign a tier at design review:
| Tier | Feature profile | Checklist sections required | Testing depth |
|---|---|---|---|
| 1 Low | No tools, no RAG over private data, output shown only to the requesting user, no personal data | 6.1, 6.2, 6.6, 6.7 | Automated injection and output-encoding tests |
| 2 Medium | RAG over internal data, or personal data in prompts | All of 6.1 to 6.7 except 6.4 | Plus manual adversarial testing of retrieval |
| 3 High | Tools that read or write systems, external sending, or regulated data | All sections | Plus independent red team before launch |
6. Pre-launch checklist
6.1 Prompt injection and model behaviour (LLM01, LLM07)
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.1.1 | A threat model lists every source of untrusted content the model reads, including indirect sources (documents, email, web, tool output). | Must | Threat model document reviewed. |
| 6.1.2 | Untrusted content is clearly delimited and labelled as data in the prompt, separate from system instructions. | Must | Prompt template review. |
| 6.1.3 | The system prompt holds no secrets, credentials, internal URLs or authorisation rules; assume it will be disclosed. | Must | System prompt review; extraction test does not reveal anything sensitive. |
| 6.1.4 | No security decision depends on the model obeying instructions: access, approval and limits are enforced in application code. | Must | Design review; test that a successful jailbreak still cannot exceed the user's permissions. |
| 6.1.5 | Input and output filtering (guardrail classifier or rules) is applied for the feature's tier. | Should (Tier 1), Must (Tier 2-3) | Filter configuration and block-rate test. |
| 6.1.6 | A regression suite of direct and indirect injection cases runs in CI and on every model or prompt change. | Must | CI job evidence. |
Indirect injection is not theoretical: in the Gemini Calendar prompt injection case, instructions in a calendar invite description led the assistant to copy private meeting details into an event the sender could read. See the Prompt Injection Guide.
6.2 Output handling (LLM05, LLM09)
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.2.1 | Model output is encoded for its destination: HTML encoding for web, escaping for shells, and parameterised queries for any database use. | Must | Code review; test with output containing script tags and SQL metacharacters. |
| 6.2.2 | Markdown rendering does not auto-load remote images or links from model output without an allow-list (blocks data exfiltration through image URLs). | Must | Test response containing an external image URL is not fetched. |
| 6.2.3 | A strict Content Security Policy applies to pages that render model output. | Must | Response headers. |
| 6.2.4 | Where output drives code (JSON, function arguments), it is validated against a schema before use and rejected if invalid. | Must | Schema validation tests. |
| 6.2.5 | Users are told the output is AI-generated and may be wrong; high-impact answers cite sources or require human review. | Should | UI review. |
6.3 Sensitive data and RAG access (LLM02, LLM08)
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.3.1 | Retrieval enforces the requesting user's permissions at query time; the vector store does not return chunks the user could not open in the source system. | Must | Test with two users of different rights; the lower-privileged user retrieves nothing restricted. |
| 6.3.2 | Multi-tenant vector stores partition data per tenant (separate index, namespace or enforced tenant filter). | Must | Cross-tenant retrieval test. |
| 6.3.3 | Ingestion pipelines validate and authenticate document sources, and record provenance per chunk. | Must | Pipeline review. |
| 6.3.4 | Personal and secret data is minimised or redacted before prompts are sent and before content is embedded. | Must | Redaction test with seeded test data. |
| 6.3.5 | Provider terms confirm whether prompts are retained or used for training; retention matches [Organisation]'s data policy. | Must | Contract or provider setting recorded. |
| 6.3.6 | Conversation history from one user is never available in another user's context. | Must | Session isolation test. |
See the Permission-Aware RAG Guide and the RAG Security Guide.
6.4 Tools and agency (LLM06)
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.4.1 | Only the tools the feature needs are exposed, each with the narrowest functionality (no open-ended shell, SQL or URL fetch tools). | Must | Tool inventory reviewed. |
| 6.4.2 | Tools act in the end user's security context (delegated, scoped tokens), not with a shared service account holding broader rights. | Must | Credential review; test that the tool cannot act beyond the user's rights. |
| 6.4.3 | High-risk actions require explicit user confirmation showing exactly what will happen. | Must | UI test. |
| 6.4.4 | Every tool call is authorised in the downstream system (complete mediation), not only in the LLM layer. | Must | Downstream access control test. |
| 6.4.5 | Tool credentials are stored in a secrets manager, short-lived where possible, and never placed in prompts. | Must | Secret scan of prompts and logs. |
See the OWASP LLM Top 10 LLM06 Excessive Agency Guide.
6.5 Supply chain and model provenance (LLM03, LLM04)
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.5.1 | Model provider, model version and SDKs are recorded in an inventory or AI bill of materials; model versions are pinned. | Must | Inventory entry. |
| 6.5.2 | Self-hosted or downloaded models come from verified sources and are checked by hash or signature; unsafe serialisation formats are avoided or scanned. | Must (if self-hosted) | Hash check in deployment pipeline. |
| 6.5.3 | Fine-tuning and RAG datasets are versioned and their sources vetted. | Should | Dataset version record. |
| 6.5.4 | Provider API keys are scoped per application and environment, rotated, and monitored for abuse. | Must | Key inventory and usage alert. |
Stolen provider keys are a direct cost and data risk; see the LLMjacking Guide.
6.6 Abuse and cost limits (LLM10)
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.6.1 | Input size and output token limits are set per request. | Must | Oversized request is rejected. |
| 6.6.2 | Rate limits apply per user and per API key, for example [50 requests per minute], with a spend cap and alert at provider level. | Must | Load test; provider budget alert configured. |
| 6.6.3 | Timeouts and loop limits stop runaway tool chains. | Must | Test with a recursive tool scenario. |
| 6.6.4 | Logits and log probabilities are not exposed to users unless required. | Should | API response review. |
6.7 Logging, monitoring and response
| # | Requirement | Level | Verify |
|---|---|---|---|
| 6.7.1 | Prompts, retrieved document IDs, tool calls, filter decisions and responses are logged with user and session IDs, with personal data handled under [Retention Policy]. | Must | Sample trace in the log platform. |
| 6.7.2 | Alerts fire on guardrail block spikes, system prompt extraction attempts, unusual tool use and token spend anomalies. | Must | Test each alert. |
| 6.7.3 | A kill switch can disable the feature, a tool or a data source without a redeploy. | Must | Switch exercised in staging. |
| 6.7.4 | Users can report harmful or wrong output, and reports reach [Product owner]. | Should | Feedback channel tested. |
See the LLM Observability Guide.
7. Framework mapping
| Checklist section | OWASP LLM Top 10 2025 | MITRE ATLAS | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 |
|---|---|---|---|---|
| 6.1 Prompt injection | LLM01 Prompt Injection, LLM07 System Prompt Leakage | AML.T0051 LLM Prompt Injection (AML.T0051.001 Indirect), AML.T0054 LLM Jailbreak, AML.T0056 Extract LLM System Prompt | SI-10 | PR.AA-05 |
| 6.2 Output handling | LLM05 Improper Output Handling, LLM09 Misinformation | None specific | SI-15 | PR.DS-01 |
| 6.3 Data and RAG | LLM02 Sensitive Information Disclosure, LLM08 Vector and Embedding Weaknesses | AML.T0057 LLM Data Leakage | AC-6 | PR.AA-05, PR.DS-01 |
| 6.4 Tools | LLM06 Excessive Agency | AML.T0051 | AC-6 | PR.AA-05 |
| 6.5 Supply chain | LLM03 Supply Chain, LLM04 Data and Model Poisoning | None specific | SR-3 | GV.SC-01 |
| 6.6 Abuse and cost | LLM10 Unbounded Consumption | AML.T0024 Exfiltration via AI Inference API | SC-5 | DE.CM-09 |
| 6.7 Logging | All | All | AU-2 | DE.CM-09 |
Organisations running an AI management system under ISO/IEC 42001 can file the completed checklist as evidence of AI system risk treatment.
8. Exceptions
- An unmet "must" item needs a written exception naming the feature, item, compensating control, risk owner and an expiry of no more than [90 days].
- No exception is allowed for 6.1.4 (security outside the model) or 6.3.1 (retrieval permission enforcement) on Tier 2 or 3 features.
9. Compliance and measurement
- Launch coverage: share of LLM features launched with a completed checklist, target [100%].
- Injection regression pass rate: per release, with any drop investigated before release.
- Unauthorised retrieval tests: cross-user and cross-tenant tests failing safely, target 100%.
- Guardrail block rate and false positives: trended weekly per feature.
- Token spend per user: with alerts on outliers.
- Open exceptions: number and age.
10. Related documents and review
Review this checklist every [6 months], when OWASP publishes a new edition of the LLM Top 10, and after any AI security incident. Related documents: [AI Acceptable Use Policy], [Secure Development Standard], [Data Classification Policy], [Incident Response Plan].
Implementation guidance
- Tier the feature at design review and agree which sections apply before code is written.
- Design the trust boundaries first: where untrusted content enters, where permissions are checked, which tools exist. Items 6.1.4, 6.3.1 and 6.4.2 are architecture decisions that are hard to retrofit.
- Build the test suite alongside the feature: injection cases, cross-user retrieval, output encoding and oversized inputs, run in CI.
- Launch behind a flag to a pilot group with full logging and the kill switch tested; review logs and feedback for [2 weeks].
- Red team Tier 3 features before general availability; see the AI Red Teaming Guide.
- Roll back by disabling the flag, a tool or a data source; keep the previous prompt and model version deployable.
Small teams using a hosted model through an API can meet most Tier 1 items with output encoding, rate limits, a spend cap and an injection test set. Larger organisations should centralise guardrails, key management and logging in an AI gateway so each team inherits them; see the AI Gateway and LLM Gateway Guide and the AI Guardrails and AI Firewalls Guide.
Source baselines
- OWASP: Top 10 for LLM Applications 2025
- OWASP: LLM Applications Cybersecurity and Governance Checklist
- NCSC: Guidelines for secure AI system development
- NIST: AI 600-1 Generative Artificial Intelligence Profile
- MITRE ATLAS: LLM Prompt Injection
- OWASP: Application Security Verification Standard 5.0
- NIST: SP 800-53 Rev. 5
This template is provided for guidance and must be adapted to your organisation. Related NHI Mgmt Group resources: OWASP Top 10 for LLM Applications Guide · LLM Security Guide · Top 10 LLM Deployment Mistakes · Prompt Injection Guide