AI model risk indices expose gaps in GenAI security testing

At a glance

What this is: This is Lakera’s analysis of a runtime-focused AI model risk benchmark that measures how LLMs resist direct and indirect adversarial attacks in realistic enterprise scenarios.

Why it matters: It matters because security, IAM, and governance teams need evidence of model behavior under pressure before they trust GenAI in production, especially where models interact with sensitive data, tools, and business workflows.

👉 Read Lakera's analysis of the AI Model Risk Index for GenAI security

Context

AI model risk is no longer a theoretical concern once GenAI moves into production. Static evaluation methods often miss the gap between controlled test prompts and how a model behaves when attackers actively try to manipulate its instructions, its context, or the data it processes.

For identity and governance teams, the issue is not just model quality. It is whether the operating model around the system can detect, measure, and limit bad behavior when the model is exposed to real adversarial pressure in direct user interactions and in tool- or retrieval-based workflows.

Key questions

Q: How should security teams evaluate GenAI models before production?

A: Security teams should test models with realistic adversarial scenarios, including direct prompt attacks and indirect instruction injection through retrieved content. The goal is to measure whether the model maintains its intended behavior under pressure. Approval should depend on repeatable evidence, not on a one-time benchmark score or vendor assurance.

Q: Why do static prompt benchmarks fail for enterprise LLM governance?

A: Static prompt benchmarks usually measure responses to a fixed set of inputs, which misses how attackers exploit context, hidden instructions, and workflow-specific assumptions. Enterprise governance needs to know whether the model stays within bounds when it is embedded in real systems. That makes runtime testing more relevant than isolated prompt review.

Q: What should teams do about indirect prompt injection in RAG systems?

A: Teams should treat retrieved content as untrusted until it is filtered, validated, or constrained. Indirect prompt injection works because the model processes instructions embedded in normal business data. Defenses should combine source controls, content inspection, and limited tool authority so the model cannot act on hidden commands.

Q: How do organizations use AI risk scores in governance decisions?

A: Organizations should use AI risk scores as one input to model approval, exception handling, and periodic reassessment. The score becomes valuable when it is repeatable and tied to deployment context. It should sit alongside security review evidence, not replace it, because different models fail in different ways.

Technical breakdown

Why static prompt tests miss real model risk

Traditional LLM testing often checks a fixed set of prompts and reviews outputs for obvious failures. That approach measures surface behavior, not resilience under attack. A model can look safe in a lab and still break mission-specific rules when an attacker changes the prompt shape, hides instructions in retrieved content, or pushes the model into boundary cases the benchmark never exercised. Runtime-focused evaluation is different because it tests whether the model keeps its intended role when conditions become adversarial. That matters for production because the security question is not only what the model says, but whether it can be steered into doing what it was never supposed to do.

Practical implication: teams should evaluate models against attack scenarios, not just static prompt suites.

Direct and indirect attacks create different failure modes

Direct attacks target the model through user input, usually by trying to override instructions, reveal hidden context, or force policy evasion. Indirect attacks are more subtle. They embed malicious instructions in content the model later processes, such as documents, tickets, or retrieved web pages. That means the model may be compromised without ever receiving an obviously hostile prompt. These two paths matter because the defensive controls are different: one concerns user interaction boundaries, the other concerns what the model is allowed to consume and trust. In enterprise settings, indirect attacks are especially important in retrieval-augmented generation and tool-using workflows.

Practical implication: security teams need separate controls for user input abuse and instruction poisoning in retrieved or processed content.

Risk scores turn model behavior into governance data

A 0 to 100 risk score is valuable because it converts qualitative uncertainty into a repeatable governance signal. It lets teams compare models, monitor changes over time, and tie deployment decisions to measurable behavior rather than vendor claims. The point is not that a single score can capture all risk. The point is that governance cannot operate on anecdotes when models are being embedded into business processes. For IAM, security architecture, and compliance teams, a standardized score becomes part of evidence collection, model selection, and exception handling. It gives reviewers something concrete to inspect when deciding whether a model is ready for production use.

Practical implication: build model approval and review processes around repeatable risk evidence, not one-time assessment narratives.

NHI Mgmt Group analysis

Static benchmark culture is the wrong control model for GenAI. Lakera’s core point is that models must be evaluated under adversarial pressure, not only against fixed prompt lists. That shift matters because real attackers do not behave like test suites, and enterprise models do not operate in isolated lab conditions. The practical conclusion is that governance has to measure behavioral resilience, not only content safety.

Instruction poisoning is now a first-class enterprise risk in retrieval and tool-using systems. Indirect attacks exploit the fact that models trust the content they process unless explicitly constrained. That changes the risk model for RAG, support automation, and agentic workflows because malicious instructions can arrive through ordinary business data. Practitioners should treat processed content as a governance surface, not just an input source.

Risk scoring becomes useful only when it informs deployment decisions. A standardized score can compare models, but it also exposes how uneven model failure modes are across attack types. Some models fail by refusing legitimate work, others by following malicious instructions, and some do both. That means the control problem is not model selection alone, but matching model behaviour to the acceptable risk envelope of the use case.

Behavioral assurance for GenAI sits at the intersection of NHI governance and AI risk management. Once a model can interact with tools, data, and workflows, the security question is no longer purely about prompts. It is about which systems the model can influence, what evidence proves safe operation, and how review cycles capture change over time. The practitioner takeaway is that GenAI governance must be operational, measurable, and repeatable.

Model risk indices create a named governance concept worth tracking: runtime model resilience. This is the ability of a model to maintain intended behavior while exposed to direct or indirect attack pressure in a real deployment context. That concept is more useful than generic “AI safety” because it ties the benchmark to observable enterprise failure modes. Practitioners should use runtime model resilience as the review lens for production readiness.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, according to The State of Secrets in AppSec.
• Only 44% of developers are reported to follow security best practices for secrets management, according to The State of Secrets in AppSec.
• For a broader governance lens, Top 10 NHI Issues shows the identity-control patterns that repeatedly create avoidable exposure.

What this signals

Lakera’s benchmark is a reminder that GenAI governance will increasingly depend on evidence of behaviour under attack, not only on policy statements or red-team narratives. As model use expands, the control question shifts toward whether teams can prove runtime model resilience before a model is allowed into workflows that touch data, tools, or decisions.

Runtime model resilience: this is the capability to preserve intended behaviour when the model is exposed to direct or indirect adversarial pressure. For practitioners, that means reassessing model approval gates, fallback logic, and incident triage paths as a single control system, not separate review exercises.

The governance signal is broader than LLM security alone. When models sit inside retrieval, automation, and identity-adjacent workflows, the review problem starts to look like lifecycle management for non-human decision points, which is why framework alignment matters alongside technical testing.

For practitioners

• Adopt adversarial model testing Evaluate LLMs with direct prompt abuse, indirect instruction injection, and task-specific guardrails before production approval. Use scenarios that mirror real business workflows rather than fixed generic prompts.
• Separate controls for retrieved content and user prompts Treat documents, tickets, search results, and other processed content as a distinct trust domain. Apply filtering, source validation, and constraint checks before the model can act on hidden instructions.
• Tie approvals to measurable risk scores Require a repeatable risk score or equivalent evidence package for model selection, exception handling, and periodic reassessment. Make the score part of the review record so changes in model behavior are visible over time.
• Test fallback and containment paths Verify how the system behaves when the model refuses legitimate requests, follows malicious instructions, or degrades under attack. Ensure human review, fallback responses, and access constraints are available before users encounter failure.

Key takeaways

• LLM governance fails when teams rely on static prompt testing instead of adversarial evaluation under realistic conditions.
• Direct attacks and indirect instruction injection expose different weaknesses, so model controls must address both input abuse and processed content trust.
• Risk scores are only useful when they feed model approval, exception handling, and ongoing reassessment in production.

Key terms

• Adversarial evaluation: Adversarial evaluation is the practice of testing a model against hostile inputs designed to make it fail, misbehave, or reveal hidden instructions. For GenAI, this means assessing behavior under realistic attack conditions rather than only checking whether outputs look safe in ordinary use.
• Indirect prompt injection: Indirect prompt injection occurs when malicious instructions are hidden inside content a model later processes, such as documents, web pages, or support tickets. The model may follow those instructions without the attacker ever interacting with it directly, which makes retrieval and tool workflows especially exposed.
• Runtime model resilience: Runtime model resilience is the ability of a model to keep its intended behavior while facing direct or indirect adversarial pressure in production-like conditions. It is an operational measure of whether the model can hold boundaries when context, tools, or instructions are manipulated.

What's in the full article

Lakera's full article covers the operational detail this post intentionally leaves for the source:

• The benchmark categories and scoring logic used to compare model behavior across attack types
• Examples of how direct and indirect attacks produce different failure patterns in practice
• The applied use cases behind the evaluation, including RAG and code generation contexts
• How security teams can use the index to support deployment decisions and governance reviews

👉 Lakera's full post covers the benchmark design, attack categories, and model scoring examples in more detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.