An LLM application that passed its launch review drifts every time the model, the system prompt, the retrieval corpus or the guardrail policy changes, and most teams never measure whether it is still as safe as the day it shipped. This health check is a recurring review of large language model (LLM) applications already in production: chatbots, copilots, summarisers and retrieval-augmented (RAG) assistants. It gives nine metrics with formulas and thresholds covering red team coverage, prompt injection resistance, guardrail behaviour, sensitive output leakage, version control, retrieval access, evaluation regressions, open jailbreak findings and logging. It is for product security engineers, AI platform owners and the people who sign off AI risk each quarter.
This page measures after release; it does not repeat the launch controls. For the pre-release sign-off list, use the LLM Application Security Checklist. If the application can call tools or act autonomously, read Agentic AI Security vs LLM App Security first: agent permissions need their own review.
What this health check covers
- In scope: every production feature that sends prompts to a hosted or self-hosted model and uses the response, its prompt templates, guardrail policies, retrieval indexes, evaluation suites and logs.
- Who runs it: the [AI platform owner] collects the data; [Product security] validates the injection, leakage and retrieval metrics; the [CISO] or [Head of AI governance] signs off.
- What it is not: a red team exercise. It measures whether testing happened and what it found. For running the tests, use the AI Red Teaming Guide.
Key takeaways
- Re-test on every change, not every year. Model upgrades, prompt edits and corpus changes are the events that break defences; tie the injection suite and evaluations to them.
- Treat the guardrail as a sensor with error rates. Track block rate, false positives and misses together; a block rate alone says nothing about safety.
- Sample production, not just test sets. Leakage rate comes from reviewing real responses, which is also what NIST AI 600-1 suggests for measuring harmful content in deployment.
- Retrieval permissions are the leakage control that matters most. Measure, with test identities, whether the RAG layer returns content a user could not open directly.
- Old jailbreak findings are open doors. Age of open findings shows whether the organisation fixes what its testers find.
How to use this template
Copy the metrics table and findings log into your tracker and run the review per application at the cadence below. Score each metric green, amber or red, record every amber or red result as a finding with an owner and due date, and keep each completed review for trend comparison. Replace values in square brackets with your own.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| Injection regression suite and evaluations | On every model, prompt, guardrail or corpus change, and at least weekly | [Engineering lead] | [Product security] | Suite results per release, linked to the change record |
| Production output sampling for leakage | Monthly | [Product security] | [Data protection officer] | Sample worksheet, findings |
| Full metrics review (this page) | Quarterly per Tier 2 and Tier 3 application, half-yearly per Tier 1 | [AI platform owner] | [CISO] and [Head of AI governance] | Metrics table, extracts, findings log |
| Independent red team | At least annually for Tier 3, and after major architecture changes | [AI security lead] | [CISO] | Test report, findings in tracker |
Tiers follow the risk tiering in the LLM Application Security Checklist: Tier 1 has no tools and no private retrieval, Tier 3 reads or writes systems or handles regulated data.
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Change" means a new model version, a prompt template edit, a guardrail policy edit or a retrieval corpus or index change.
| ID | Metric | Formula | Data source | Green | Amber | Red |
|---|---|---|---|---|---|---|
| LLM-01 | Red team coverage | Production LLM applications with adversarial testing in the last [12] months (Tier 3: last [6] months), covering direct and indirect injection, ÷ production LLM applications | AI inventory, test reports | 100% | 80% to below 100% | Below 80% |
| LLM-02 | Prompt injection test pass rate | Injection regression cases resisted ÷ cases run, on the latest release, reported separately for direct and indirect cases | CI or evaluation pipeline results | At least 98%, no failed case that breaches data or permissions | 90% to below 98% | Below 90%, or any failed case that reaches data or tools |
| LLM-03 | Guardrail block rate and error | Blocks ÷ requests, tracked weekly; plus sampled blocks judged wrong ÷ blocks sampled (minimum [50] per quarter) | Guardrail verdict logs, reviewer worksheet | Block rate stable within [±50%] of baseline; false positives below 5% | Unexplained shift, or false positives 5% to 15% | Guardrail disabled on any route, or error not measured |
| LLM-04 | Sensitive output leakage rate | Sampled production responses containing personal data, secrets, system prompt content or other users' data ÷ responses sampled | Response logs, pattern scan plus manual review | 0% | Isolated case, contained and fixed | Any repeat, or any cross-user disclosure |
| LLM-05 | Model and prompt version control | Production model, system prompt, guardrail policy and index versions that match an approved, recorded release ÷ components deployed | Deployment records, prompt registry, configuration export | 100% | Untracked change found and recorded within the period | Production prompts edited outside change control |
| LLM-06 | RAG source access enforcement | Retrieval probes by test identities that returned no unauthorised chunk ÷ probes run | Permission test harness, retrieval logs | 100%, with every production index probed | 100%, but some indexes not probed this period | Any unauthorised chunk returned |
| LLM-07 | Eval regression rate | Changes released with a security or safety evaluation score below the previous release ÷ changes released | Evaluation pipeline history | 0% without approved exception | Regression released with exception and fix date | Regression released with no evaluation run |
| LLM-08 | Jailbreak finding age | Median and maximum age in days of open jailbreak and injection findings, by severity | Findings tracker | High closed within [30] days | High open 30 to 60 days | High open over 60 days, or critical over 14 |
| LLM-09 | Logging coverage | Production LLM applications logging prompts, responses, guardrail verdicts, retrieval sources and model version, with redaction and retention set, sent to the SIEM ÷ production LLM applications | Log pipeline configuration, SIEM source list | 100% | 90% to below 100% | Below 90%, or logs holding unredacted secrets |
Framework mapping
| Metric | OWASP Top 10 for LLM Applications 2025 | MITRE ATLAS | NIST AI 600-1 suggested action |
|---|---|---|---|
| LLM-01, LLM-02, LLM-08 | LLM01 Prompt Injection (prevention: regular adversarial testing, treating the model as an untrusted user) | AML.T0051 LLM Prompt Injection (.000 Direct, .001 Indirect), AML.T0054 LLM Jailbreak | MS-2.7-007 red-teaming for prompt injection; MS-2.6-007 regular evaluation of whether safety measures can be circumvented |
| LLM-03 | LLM01 (input and output filtering) | AML.T0054 | MS-2.5-006 regular review of security and safety guardrails |
| LLM-04 | LLM02 Sensitive Information Disclosure, LLM07 System Prompt Leakage | AML.T0057 LLM Data Leakage, AML.T0056 Extract LLM System Prompt | MS-2.10-001 red-teaming for revealed personal or sensitive information; MS-2.11-002 sampling live traffic for review |
| LLM-05, LLM-07 | LLM03 Supply Chain, LLM04 Data and Model Poisoning | AML.T0070 RAG Poisoning | MS-2.8-003 version control across the lifecycle; MS-2.7-009 regular verification that security measures remain effective |
| LLM-06 | LLM08 Vector and Embedding Weaknesses (permission-aware vector stores) | AML.T0057 | MS-2.7-004 tracking unauthorised access and extraction attempts |
| LLM-09 | LLM08 (detailed immutable retrieval logs) | Supports detection of all of the above | GV-1.5-003 retention of testing and evaluation records |
OWASP is explicit that the system prompt should not be treated as a secret or used as a security control, which is why LLM-04 counts system prompt content as leakage only when it carries data that should never have been there. In NIST CSF 2.0 terms the review supports ID.IM-02 (improvements from security tests and exercises) and DE.CM-09 (monitoring of computing software and runtime environments); in NIST SP 800-53 Rev. 5 it draws on CA-7 Continuous Monitoring, CA-8 Penetration Testing, CM-3 Configuration Change Control and AU-12 Audit Record Generation.
How to collect the data
- Start from the AI inventory. List every production LLM feature with its tier, model, owner and whether it uses retrieval. LLM-01 and LLM-09 are only as good as this list; reconcile it with model API traffic from your gateway or egress logs.
- Pull the regression suite results for the latest release of each application. Count direct and indirect cases separately: indirect cases (instructions hidden in documents, web pages or tool output) usually fail first. See the Prompt Injection Guide for case design.
- Sample guardrail blocks each quarter; two reviewers judge whether each block was correct. Compare weekly block counts with the baseline and explain any step change: a sudden drop can mean a policy was weakened, a sudden rise can mean an attack campaign or a broken classifier.
- Sample production responses each month, weighted to Tier 3. Scan for secret and personal data patterns, then review a random subset manually for system prompt content and other users' data.
- Compare deployed versions with the release record for the model identifier, system prompt hash, guardrail policy version and index build. Any mismatch is a change made outside control.
- Run retrieval probes with two or more test identities holding different permissions, asking for content only one may see. See the Permission-Aware RAG Guide and the RAG Security Test Plan Template.
- Export evaluation history to compare scores release on release, and the findings tracker for LLM-08.
- Check the log pipeline per application against the fields listed in LLM-09. The LLM Observability Guide covers collection.
Red flags that need action now
- A response in production disclosed another user's data or a credential.
- A retrieval probe returned content the test identity could not open in the source system.
- The production system prompt or model version differs from the last approved release.
- An indirect injection case that reaches data or tools passes in CI and the release shipped anyway.
- A guardrail policy was turned off on a production route with no change record.
- Prompt and response logs contain unredacted secrets.
Treat a confirmed leak or a successful injection in production as an incident, not a metric: follow the Prompt Injection Incident Response Playbook.
Common failure patterns
- The injection suite runs on the old model. The provider updated the model alias, the application picked it up, and nobody re-ran the tests. Pin model versions and trigger the suite on change.
- Block rate reported as a success metric. A falling block rate was celebrated while a policy edit had quietly disabled a category.
- Leakage measured only in test. Production users ask questions no test set anticipated; sample real responses.
- Retrieval permissions checked at ingestion only. Source permissions changed, the index did not; probes with test identities catch it.
- Findings closed by prompt edits. A new line in the system prompt hides one jailbreak and invites the next variant; fix in code or policy outside the model, then re-test.
Findings log
| ID | Finding | Application | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|---|
| HC-LLM-01 | [Description, population and figure] | [Application] | [LLM-0x] | [Critical / High / Medium / Low] | [Role] | [Date] | [Open / In progress / Closed / Risk accepted] |
| HC-LLM-02 | [Description] | [Application] | [Metric] | [Severity] | [Role] | [Date] | [Status] |
Suggested due dates: red flags within 7 days, red metrics within 30 days, amber within 90 days. Risk acceptance needs the [AI platform owner]'s signature and an expiry date.
Remediation map
| Finding type | Where to fix it |
|---|---|
| Coverage, injection and jailbreak findings (LLM-01, LLM-02, LLM-08) | AI Red Teaming Guide and OWASP LLM Top 10 LLM01 Prompt Injection Guide |
| Guardrail drift (LLM-03) | AI Guardrails and AI Firewalls Guide |
| Leakage and logging (LLM-04, LLM-09) | OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide and Generative AI Data Loss Prevention Guide |
| Version control and regressions (LLM-05, LLM-07) | Model-Level vs System-Level AI Evaluation |
| Retrieval access (LLM-06) | RAG Security Guide |
Related guides
- LLM Application Security Checklist: the pre-release controls this review assumes are in place.
- LLM Security Guide: the overview of LLM threats and controls.
- AI Security Maturity Model: where recurring measurement sits in a wider AI security programme.
- Prompt Injection Incident Response Playbook: what to do when a red flag is a live incident.
- AI gateway health check: the neighbouring review that shares gateway logs and guardrail data.
References
- OWASP: Top 10 for LLM Applications 2025
- OWASP Gen AI Security Project: GenAI Red Teaming Guide
- NIST: AI 600-1 Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile
- NIST: AI 100-2 E2025 Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations
- MITRE ATLAS: LLM Prompt Injection
- NCSC: Guidelines for secure AI system development
- NIST: Cybersecurity Framework 2.0
Related NHI Mgmt Group resources: LLM Application Security Checklist, LLM Security Guide, OWASP Top 10 for LLM Applications Guide, AI gateway health check.