How should security teams test LLM applications for runtime security failures?

Test the full runtime path, not just the model in isolation. Use adversarial prompts, indirect injection through retrieved content, and tool-call scenarios to see whether the application can be steered into unsafe behaviour once retrieval, agents, and external APIs are active. That is where most LLM risk actually appears.

Testing the runtime path, not the base model

LLM applications fail most often at runtime boundaries, where prompts, retrieved content, tool outputs, and agent actions combine into one execution path. A model that appears well behaved in isolation can still be steered into unsafe disclosures, policy bypass, or harmful tool use once the application starts merging user input with external context. For security teams, the real test is whether the application preserves trust boundaries after retrieval augmentation, function calling, and orchestration are enabled. OWASP’s OWASP Top 10 for Agentic Applications 2026 is a useful reference because it focuses on the application layer where those failures emerge. In practice, many teams discover the weakness only after an assistant has already accepted untrusted context as instructions rather than data.

How to exercise prompts, retrieval, and tool calls together

Runtime testing should emulate the full chain the application actually uses. Start with direct adversarial prompts to see whether the system resists obvious unsafe requests, then add indirect injection through documents, tickets, web pages, chat logs, or other retrieved sources to check whether hostile instructions survive ingestion. After that, test tool-call scenarios, because the question is not only whether the model can be persuaded, but whether it can be persuaded to act through an API, connector, or workflow with real side effects.

A practical test set should cover at least three conditions:

user-only prompts that try to override policy or extract secrets
retrieved content that contains conflicting instructions or hidden malicious intent
agentic workflows where the model can call tools, write records, or trigger downstream actions

This matters because many failures are compositional. A retrieval layer may surface tainted context, the prompt may merge that context without clear delimiting, and the tool layer may then execute the model’s mistaken interpretation as if it were trustworthy. NIST’s NIST AI 600-1 Generative AI Profile is relevant here because it frames generative AI risk as an application and lifecycle problem, not only a model-quality problem. Testing should therefore verify boundary handling, refusal behaviour, escalation paths, and whether sensitive actions require human confirmation. Where applications use external APIs, the most important question is whether the model can turn a language vulnerability into a control failure. The guidance breaks down when the test environment omits the actual retrieval sources, tool permissions, or orchestration logic used in production.

Where LLM runtime testing stops being a simple prompt-review exercise

Tighter runtime controls often reduce model flexibility, so organisations have to balance safer behaviour against user experience and automation value. The standard answer becomes less reliable when the application has multiple context sources, long-lived memory, or agents that can chain actions across systems. In those cases, a harmless-looking prompt test may miss the real failure mode because the dangerous instruction arrives indirectly, survives summarisation, or becomes more powerful after a tool response is fed back into the conversation.

This is one place where guidance differs across the field. Some teams treat prompt injection as the primary concern, while others focus more on tool misuse and delegated authority. In practice, both can matter, but the dominant failure depends on whether the application is chat-only, retrieval-heavy, or agentic. If the system can act on behalf of a user, then runtime testing should also check whether action boundaries, approval requirements, and content provenance survive across turns. For broader AI governance context, the NIST AI Risk Management Framework is useful, but it does not replace application-specific abuse testing. The same runtime test plan should also include negative cases for overblocking, because controls that stop attacks by breaking normal workflows are usually not fit for production.

Risk and Threat Considerations

Runtime failure in LLM applications creates a direct exposure path from untrusted text to unsafe action. The risk is not confined to incorrect answers; it includes prompt injection, data exfiltration, unauthorized tool execution, and loss of control over agent behaviour once external context is treated as instruction.

Failure mechanism: An attacker or hostile document can place malicious instructions in retrieved content, surrounding text, or tool output. If the application fails to preserve trust boundaries, the model may follow those instructions, reveal sensitive context, or invoke connected systems in ways the operator did not intend.

Impact: The consequence can be disclosure of secrets, manipulation of records, unsafe external actions, or an agent that becomes unreliable as a decision support layer because its runtime behaviour is no longer governed by the original user request.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Agentic AI Top 10	A1	Tests how runtime inputs can override intended instructions.
Recommendation: Validate that untrusted content cannot redirect agent behaviour or tool use.
MITRE ATLAS	ATLAS-000	Covers attack patterns against AI systems and their runtime surfaces.
Recommendation: Map tests to known AI abuse patterns across prompts, retrieval, and actions.
NIST AI RMF	GOVERN	Runtime testing supports AI governance, accountability, and lifecycle oversight.
Recommendation: Treat runtime abuse testing as part of governed AI risk management.
NIST AI 600-1	MAP	Generative AI risk must be assessed in application and deployment context.
Recommendation: Assess model behaviour in the live system, not only in isolation.
CSA MAESTRO	TM-1	Agentic workflows need threat modelling across tools, memory, and connectors.
Recommendation: Model abuse paths where context, orchestration, and tools combine.

Practitioner Guidance

What to prioritise: Test the highest-privilege path first. If the application can read internal sources or call tools that modify systems, that path deserves more attention than a chat-only interface because the blast radius is materially larger.

What to verify: Verify that untrusted retrieved text stays untrusted all the way through the request lifecycle. The key check is whether the application distinguishes instructions from data after retrieval, summarisation, and tool feedback, not just before the prompt reaches the model.

Decision rule: If a runtime test only proves the model refuses an obvious malicious prompt, treat that as incomplete. The control is only credible when it also resists indirect injection and does not perform an unsafe tool action after being nudged through context.

Practitioner takeaway: The most important judgment is whether the application can still enforce policy after the model starts interacting with real data and real tools, because that is where LLM security usually changes from content moderation into access control.