Join our Newsletter — 33% off our NHI Course
Home› Guides› LLM Application Security Health Check: 9 Metrics for…
Template LLM & GenAI Security

LLM Application Security Health Check: 9 Metrics for Prompt Injection and Data Leakage

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 11 min read
On this page

An LLM application that passed its launch review drifts every time the model, the system prompt, the retrieval corpus or the guardrail policy changes, and most teams never measure whether it is still as safe as the day it shipped. This health check is a recurring review of large language model (LLM) applications already in production: chatbots, copilots, summarisers and retrieval-augmented (RAG) assistants. It gives nine metrics with formulas and thresholds covering red team coverage, prompt injection resistance, guardrail behaviour, sensitive output leakage, version control, retrieval access, evaluation regressions, open jailbreak findings and logging. It is for product security engineers, AI platform owners and the people who sign off AI risk each quarter.

This page measures after release; it does not repeat the launch controls. For the pre-release sign-off list, use the LLM Application Security Checklist. If the application can call tools or act autonomously, read Agentic AI Security vs LLM App Security first: agent permissions need their own review.

What this health check covers

  • In scope: every production feature that sends prompts to a hosted or self-hosted model and uses the response, its prompt templates, guardrail policies, retrieval indexes, evaluation suites and logs.
  • Who runs it: the [AI platform owner] collects the data; [Product security] validates the injection, leakage and retrieval metrics; the [CISO] or [Head of AI governance] signs off.
  • What it is not: a red team exercise. It measures whether testing happened and what it found. For running the tests, use the AI Red Teaming Guide.

Key takeaways

  • Re-test on every change, not every year. Model upgrades, prompt edits and corpus changes are the events that break defences; tie the injection suite and evaluations to them.
  • Treat the guardrail as a sensor with error rates. Track block rate, false positives and misses together; a block rate alone says nothing about safety.
  • Sample production, not just test sets. Leakage rate comes from reviewing real responses, which is also what NIST AI 600-1 suggests for measuring harmful content in deployment.
  • Retrieval permissions are the leakage control that matters most. Measure, with test identities, whether the RAG layer returns content a user could not open directly.
  • Old jailbreak findings are open doors. Age of open findings shows whether the organisation fixes what its testers find.

How to use this template

Copy the metrics table and findings log into your tracker and run the review per application at the cadence below. Score each metric green, amber or red, record every amber or red result as a finding with an owner and due date, and keep each completed review for trend comparison. Replace values in square brackets with your own.

Cadence and sign-off

ReviewFrequencyOwnerApproverEvidence kept
Injection regression suite and evaluationsOn every model, prompt, guardrail or corpus change, and at least weekly[Engineering lead][Product security]Suite results per release, linked to the change record
Production output sampling for leakageMonthly[Product security][Data protection officer]Sample worksheet, findings
Full metrics review (this page)Quarterly per Tier 2 and Tier 3 application, half-yearly per Tier 1[AI platform owner][CISO] and [Head of AI governance]Metrics table, extracts, findings log
Independent red teamAt least annually for Tier 3, and after major architecture changes[AI security lead][CISO]Test report, findings in tracker

Tiers follow the risk tiering in the LLM Application Security Checklist: Tier 1 has no tools and no private retrieval, Tier 3 reads or writes systems or handles regulated data.

Metrics

NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Change" means a new model version, a prompt template edit, a guardrail policy edit or a retrieval corpus or index change.

IDMetricFormulaData sourceGreenAmberRed
LLM-01Red team coverageProduction LLM applications with adversarial testing in the last [12] months (Tier 3: last [6] months), covering direct and indirect injection, ÷ production LLM applicationsAI inventory, test reports100%80% to below 100%Below 80%
LLM-02Prompt injection test pass rateInjection regression cases resisted ÷ cases run, on the latest release, reported separately for direct and indirect casesCI or evaluation pipeline resultsAt least 98%, no failed case that breaches data or permissions90% to below 98%Below 90%, or any failed case that reaches data or tools
LLM-03Guardrail block rate and errorBlocks ÷ requests, tracked weekly; plus sampled blocks judged wrong ÷ blocks sampled (minimum [50] per quarter)Guardrail verdict logs, reviewer worksheetBlock rate stable within [±50%] of baseline; false positives below 5%Unexplained shift, or false positives 5% to 15%Guardrail disabled on any route, or error not measured
LLM-04Sensitive output leakage rateSampled production responses containing personal data, secrets, system prompt content or other users' data ÷ responses sampledResponse logs, pattern scan plus manual review0%Isolated case, contained and fixedAny repeat, or any cross-user disclosure
LLM-05Model and prompt version controlProduction model, system prompt, guardrail policy and index versions that match an approved, recorded release ÷ components deployedDeployment records, prompt registry, configuration export100%Untracked change found and recorded within the periodProduction prompts edited outside change control
LLM-06RAG source access enforcementRetrieval probes by test identities that returned no unauthorised chunk ÷ probes runPermission test harness, retrieval logs100%, with every production index probed100%, but some indexes not probed this periodAny unauthorised chunk returned
LLM-07Eval regression rateChanges released with a security or safety evaluation score below the previous release ÷ changes releasedEvaluation pipeline history0% without approved exceptionRegression released with exception and fix dateRegression released with no evaluation run
LLM-08Jailbreak finding ageMedian and maximum age in days of open jailbreak and injection findings, by severityFindings trackerHigh closed within [30] daysHigh open 30 to 60 daysHigh open over 60 days, or critical over 14
LLM-09Logging coverageProduction LLM applications logging prompts, responses, guardrail verdicts, retrieval sources and model version, with redaction and retention set, sent to the SIEM ÷ production LLM applicationsLog pipeline configuration, SIEM source list100%90% to below 100%Below 90%, or logs holding unredacted secrets

Framework mapping

MetricOWASP Top 10 for LLM Applications 2025MITRE ATLASNIST AI 600-1 suggested action
LLM-01, LLM-02, LLM-08LLM01 Prompt Injection (prevention: regular adversarial testing, treating the model as an untrusted user)AML.T0051 LLM Prompt Injection (.000 Direct, .001 Indirect), AML.T0054 LLM JailbreakMS-2.7-007 red-teaming for prompt injection; MS-2.6-007 regular evaluation of whether safety measures can be circumvented
LLM-03LLM01 (input and output filtering)AML.T0054MS-2.5-006 regular review of security and safety guardrails
LLM-04LLM02 Sensitive Information Disclosure, LLM07 System Prompt LeakageAML.T0057 LLM Data Leakage, AML.T0056 Extract LLM System PromptMS-2.10-001 red-teaming for revealed personal or sensitive information; MS-2.11-002 sampling live traffic for review
LLM-05, LLM-07LLM03 Supply Chain, LLM04 Data and Model PoisoningAML.T0070 RAG PoisoningMS-2.8-003 version control across the lifecycle; MS-2.7-009 regular verification that security measures remain effective
LLM-06LLM08 Vector and Embedding Weaknesses (permission-aware vector stores)AML.T0057MS-2.7-004 tracking unauthorised access and extraction attempts
LLM-09LLM08 (detailed immutable retrieval logs)Supports detection of all of the aboveGV-1.5-003 retention of testing and evaluation records

OWASP is explicit that the system prompt should not be treated as a secret or used as a security control, which is why LLM-04 counts system prompt content as leakage only when it carries data that should never have been there. In NIST CSF 2.0 terms the review supports ID.IM-02 (improvements from security tests and exercises) and DE.CM-09 (monitoring of computing software and runtime environments); in NIST SP 800-53 Rev. 5 it draws on CA-7 Continuous Monitoring, CA-8 Penetration Testing, CM-3 Configuration Change Control and AU-12 Audit Record Generation.

How to collect the data

  1. Start from the AI inventory. List every production LLM feature with its tier, model, owner and whether it uses retrieval. LLM-01 and LLM-09 are only as good as this list; reconcile it with model API traffic from your gateway or egress logs.
  2. Pull the regression suite results for the latest release of each application. Count direct and indirect cases separately: indirect cases (instructions hidden in documents, web pages or tool output) usually fail first. See the Prompt Injection Guide for case design.
  3. Sample guardrail blocks each quarter; two reviewers judge whether each block was correct. Compare weekly block counts with the baseline and explain any step change: a sudden drop can mean a policy was weakened, a sudden rise can mean an attack campaign or a broken classifier.
  4. Sample production responses each month, weighted to Tier 3. Scan for secret and personal data patterns, then review a random subset manually for system prompt content and other users' data.
  5. Compare deployed versions with the release record for the model identifier, system prompt hash, guardrail policy version and index build. Any mismatch is a change made outside control.
  6. Run retrieval probes with two or more test identities holding different permissions, asking for content only one may see. See the Permission-Aware RAG Guide and the RAG Security Test Plan Template.
  7. Export evaluation history to compare scores release on release, and the findings tracker for LLM-08.
  8. Check the log pipeline per application against the fields listed in LLM-09. The LLM Observability Guide covers collection.

Red flags that need action now

  • A response in production disclosed another user's data or a credential.
  • A retrieval probe returned content the test identity could not open in the source system.
  • The production system prompt or model version differs from the last approved release.
  • An indirect injection case that reaches data or tools passes in CI and the release shipped anyway.
  • A guardrail policy was turned off on a production route with no change record.
  • Prompt and response logs contain unredacted secrets.

Treat a confirmed leak or a successful injection in production as an incident, not a metric: follow the Prompt Injection Incident Response Playbook.

Common failure patterns

  • The injection suite runs on the old model. The provider updated the model alias, the application picked it up, and nobody re-ran the tests. Pin model versions and trigger the suite on change.
  • Block rate reported as a success metric. A falling block rate was celebrated while a policy edit had quietly disabled a category.
  • Leakage measured only in test. Production users ask questions no test set anticipated; sample real responses.
  • Retrieval permissions checked at ingestion only. Source permissions changed, the index did not; probes with test identities catch it.
  • Findings closed by prompt edits. A new line in the system prompt hides one jailbreak and invites the next variant; fix in code or policy outside the model, then re-test.

Findings log

IDFindingApplicationMetricSeverityOwnerDue dateStatus
HC-LLM-01[Description, population and figure][Application][LLM-0x][Critical / High / Medium / Low][Role][Date][Open / In progress / Closed / Risk accepted]
HC-LLM-02[Description][Application][Metric][Severity][Role][Date][Status]

Suggested due dates: red flags within 7 days, red metrics within 30 days, amber within 90 days. Risk acceptance needs the [AI platform owner]'s signature and an expiry date.

Remediation map

Finding typeWhere to fix it
Coverage, injection and jailbreak findings (LLM-01, LLM-02, LLM-08)AI Red Teaming Guide and OWASP LLM Top 10 LLM01 Prompt Injection Guide
Guardrail drift (LLM-03)AI Guardrails and AI Firewalls Guide
Leakage and logging (LLM-04, LLM-09)OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide and Generative AI Data Loss Prevention Guide
Version control and regressions (LLM-05, LLM-07)Model-Level vs System-Level AI Evaluation
Retrieval access (LLM-06)RAG Security Guide

References

Related NHI Mgmt Group resources: LLM Application Security Checklist, LLM Security Guide, OWASP Top 10 for LLM Applications Guide, AI gateway health check.

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.